FlatFile Pipeline

This is a re-implementation of an existing pipeline developed originally by core and the webteam. The new version uses eHive, so familiarity with this system is essential, and has been written to use as little memory as possible.

The Registry File

This is the way we retrieve the database connections to work with. The registry file should specify:

Here is an example of a file for v67 of Ensembl. Note the use of the Registry object within a registry file and the scoping of the package. If you omit the -db_version parameter and only use HEAD checkouts of Ensembl then this will automatically select the latest version of the API. Any change to version here must be reflected in the configuration file.

	package Reg;
	use Bio::EnsEMBL::Registry;
	Bio::EnsEMBL::Registry->no_version_check(1);
	Bio::EnsEMBL::Registry->no_cache_warnings(1);
	{
	  my $version = 67;
	  Bio::EnsEMBL::Registry->load_registry_from_multiple_dbs(
	    {
	      -host => "mydb-1",
	      -port => 3306,
	      -db_version => $version,
	      -user => "user",
	      -NO_CACHE => 1,
	    },
	    {    
	      -host => "mydb-2",
	      -port => 3306,
	      -db_version => $version,
	      -user => "user",
	      -NO_CACHE => 1,
	    },
	  );
	}
	1;

You give the registry to the init_pipeline.pl script via the -registry option

Overriding Defaults Using a New Config File

We recommend if you have a number of parameters which do not change between releases to create a configuration file which inherits from the root config file e.g.

	package MyCnf;
	use base qw/Bio::EnsEMBL::Pipeline::Flatfile::Flatfile_conf/;
	sub default_options {
	  my ($self) = @_;
	  return {
	    %{ $self->SUPER::default_options() },
	    #Override of options
	  };
	}
	1;

If you do override the config then you should use the package name for your overridden config in the upcoming example commands.

Environment

PERL5LIB

PATH

ENSEMBL_CVS_ROOT_DIR

Set to the base checkout of Ensembl. We should be able to add ensembl-hive/sql onto this path to find the SQL directory for hive e.g.

	export ENSEMBL_CVS_ROOT_DIR=$HOME/work/ensembl-checkouts

ENSADMIN_PSW

Give the password to use to log into a database server e.g.

	export ENSADMIN_PSW=wibble

Command Line Arguments

Where Multiple Supported is supported we allow the user to specify the parameter more than once on the command line. For example species is one of these options e.g.

-species human -species cele -species yeast
Name TypeMultiple Supported DescriptionDefault Required
-registryStringNoLocation of the Ensembl registry to use with this pipeline-YES
-base_pathStringNoLocation of the dumps-YES
-pipeline_db -host=StringNoSpecify a host for the hive database e.g. -pipeline_db -host=myserver.mysqlSee hive generic configYES
-pipeline_db -dbname=StringNoSpecify a different database to use as the hive DB e.g. -pipeline_db -dbname=my_dumps_testUses pipeline name by defaultNO
-speciesStringYesSpecify one or more species to process. Pipeline will only consider these species-NO
-typesStringYesSpecify each type of dump you want to produce. Supported values are embl and genbankAllNO
-db_typesStringYesThe database types to use. Supports the normal db adaptor groups e.g. core, otherfeatures etc.coreNO
-pipeline_nameStringNoName to use for the pipelineflatfile_dump_$releaseNO
-emailStringNoEmail to send pipeline summaries to upon its successful completion$USER@sanger.ac.ukNO

Example Commands

To load use normally:

	init_pipeline.pl Bio::EnsEMBL::Pipeline::PipeConfig::Flatfile_conf \
	-pipeline_db -host=my-db-host -base_path /path/to/dumps -registry reg.pm

Run a subset of species (no forcing & supports registry aliases):

	init_pipeline.pl Bio::EnsEMBL::Pipeline::PipeConfig::Flatfile_conf \
	-pipeline_db -host=my-db-host -species anolis -species celegans -species human \
	-base_path /path/to/dumps -registry reg.pm

Dumping just EMBL data (no genbank):

	init_pipeline.pl Bio::EnsEMBL::Pipeline::PipeConfig::Flatfile_conf \
	-pipeline_db -host=my-db-host -type embl \
	-base_path /path/to/dumps -registry reg.pm

Running the Pipeline

  1. Start a screen session or get ready to run the beekeeper with a nohup
  2. Choose a dump location
  3. Use an init_pipeline.pl configuration from above
  4. Sync the database using one of the displayed from init_pipeline.pl
  5. Run the pipeline in a loop with a good sleep between submissions and redirect log output (the following assumes you are using bash)
  6. beekeeper.pl -url mysql://usr:pass@server:port/db -reg_conf reg.pm -loop -sleep 5 2>&1 > my_run.log &
  7. Wait

Running without a Pipeline

Hive gives us the ability to run any Process outside of a database pipeline
run using standaloneJob.pl. We will list some useful commands to run

Dumping a Single Species

  standaloneJob.pl Bio::EnsEMBL::Pipeline::Flatfile::DumpFile \
  -reg_conf reg.pm -debug 2 \
  -release 67 -species homo_sapiens -type embl \
  -base_path /path/to/dumps

Verification

Another pipeline is provided which can verify the files produced by this
pipeline. Nothing else other than a basic prodding of file contents is
attempted.

Running with a Pipeline

The code works with a SQLite database so you do not need a MySQL database
to schedule these jobs. You will have to schedule two pipelines; one
to work with embl and another to work with genbank.

The pipeline searches for all files matching the format *.dat.gz.

  init_pipeline.pl Bio::EnsEMBL::Pipeline::PipeConfig::FlatfileChecker_conf \
  -base_path /path/to/embl/dumps -type embl
  init_pipeline.pl Bio::EnsEMBL::Pipeline::PipeConfig::FlatfileChecker_conf \
  -base_path /path/to/genbank/dumps -type genbank

Running without a Pipeline

You can run this module without a pipeline if you need to check a single
file.

  standaloneJob.pl Bio::EnsEMBL::Pipeline::Flatfile::CheckFlatfile \
  -file /path/to/embl/dumps/homo_sapiens/Homo_sapiens.chromosome.1.dat.gz \
  -type embl