Class: Util::WorkPipes

Inherits:
Object
  • Object
show all
Defined in:
../lib/work_pipes.rb

Overview

This class separates GERET engine from the domain-specific scripts. WorkPipes manages the set of external pipes connecting the pool of GE individuals with independent worker scripts. The worker script reads the phenotype (typically the source of the program in the specific language) from the standard input, evaluates it and writes the objective value(s) (typically fitness, error or another metrics) to the standard output. The worker script can set up the evaluating environment, wait for the program source in the loop, run the compiler when the source is available, run the compiled program to evaluate it, collect the results and pass it to back through the standard output to the caller, wait for another source, etc.

Running more than one worker script processes in parallel is possible and it brings the performance gain in parallel (or distributed) systems. The assignment of the specific work to the specific script cannot be directly controlled.

The STDIN-STDOUT “protocol” is to be designed between the worker script and the domain-specific classes in the GERET. For instance the Util::Individual can be subclassed using the domain-specific grammar, producing the source texts recognizable by the worker script. The grammar produces the “END-OF-PROGRAM” marker which is parsed by the worker script; the syntax of objective values generated by the worker script is parsed by the Util::Individual’s subclass, etc.

Flushing the STDOUT (by $stdout.flush or by the analogous command in script’s language) in the worker script is necessary because the WorkPipes waits for the script’s output only for the certain time. This timeout can be set up to the sufficient value.

The script should not terminate itself, the caller terminates it at the end of the process. The termination of the worker script is considered as the error and raises the exception in the WorkPipes object.

Instance Attribute Summary (collapse)

Instance Method Summary (collapse)

Constructor Details

- (WorkPipes) initialize(cmds = nil, dest = 'parse=', src = 'phenotype')

Set up the worker script by the commands in cmds array, use the dest and src for specifying attributes. See WorkPipes#commands=, WorkPipes#destination and WorkPipes#source for details. 



35
36
37
38
39
40
41
42
43
 
# File '../lib/work_pipes.rb', line 35

def initialize cmds=nil, dest='parse=', src='phenotype'
  @pipes = []
  @commands = {}
  @destination = dest
  @source = src
  @timeout = 120
  @jobs_processed = 0
  self.commands = cmds unless cmds.nil?
end

Instance Attribute Details

- (Object) destination

The method name for storing the worker script’s output into the jobs object. WorkPipes#run uses object.send( @destination, output ) The default is ‘parse=’. 



47
48
49
 
# File '../lib/work_pipes.rb', line 47

def destination
  @destination
end

- (Object) jobs_processed

The number of individuals processed from the WorkPipes initialisation. 



59
60
61
 
# File '../lib/work_pipes.rb', line 59

def jobs_processed
  @jobs_processed
end

- (Object) source

The method for retrieving worker script’s input from the jobs object. WorkPipes#run uses input = object.send( @source ) The default is ‘phenotype’. 



51
52
53
 
# File '../lib/work_pipes.rb', line 51

def source
  @source
end

- (Object) timeout

The time (in seconds) for which the WorkPipes waits for the worker scripts. If there is no worker action (reading stdin or writing to stdout) detected for a given time, the exception is raised. 



56
57
58
 
# File '../lib/work_pipes.rb', line 56

def timeout
  @timeout
end

Instance Method Details

- (Object) close

Terminate all worker scripts. 



94
95
96
97
98
 
# File '../lib/work_pipes.rb', line 94

def close
  @pipes.each { |pipe| pipe.close }
  @pipes = []
  @commands = {}
end

- (Object) commands

Command lines of the worker scripts. 



62
63
64
 
# File '../lib/work_pipes.rb', line 62

def commands
  @pipes.map { |pipe| @commands[pipe] }
end

- (Object) commands=(cmds)

Run worker scripts using the cmds. The argument cmds is the Enumerable collection of the command lines. Each script is run in the separate process by IO.popen. 



68
69
70
71
72
73
74
75
76
 
# File '../lib/work_pipes.rb', line 68

def commands= cmds
  self.close
  cmds.each do |cmd| 
    p = IO.popen( cmd, 'r+' )
    #p.sync = true
    @pipes << p 
    @commands[ p ] = cmd
  end
end

- (Object) run(jobs)

Assing the work to the worker scripts and wait for results. The jobs argument have to be the Enumerable collection of the work objects, typically the Array of Util::Individual subclasses. The work object has to provide the input (eg. the ‘phenotype’ attribute) for the work script using WorkPipes#source and has to be able to store the work script’s output using the WorkPipes#destination method (eg. the ‘PipedIndividual#parse=’ method). The WorkPipes#run can be called more times (eg. once per population’s generation). 



84
85
86
87
88
89
90
91
 
# File '../lib/work_pipes.rb', line 84

def run jobs
  @jobs_processed += jobs.size
  if /win/ =~ RbConfig::CONFIG['host_os']
    run_select_broken jobs # IO.select is broken on windows
  else
    run_select_works jobs
  end
end