Media Mover issues

Here's an example of what is being stored in {media_mover_files}:

http://[mycloudfrontid].cloudfront.net/s3.amazonaws.com[mybucketname]/image.jpg

This path is no good. The only path I want is http://s3.amazonaws.com/mybucketname/image.jpg. The path above has concatenated S3 onto the cloudfront URL and is not using a / to separate s3.amazonaws.com from my bucket name.

I think there may be a preg_match bug in mm_s3_delete_file().

Here is the original code:

  $s3_url = variable_get('mm_s3_default_server_url', "https://s3.amazonaws.com/");
  // build a list of files we are going to delete
  if (preg_match('@^(?:https?://)?([^/]+)@i', $item['storage_file'], $s3_url)) {
    $files[] = $item['storage_file'];
  }
  if (preg_match('@^(?:https?://)?([^/]+)@i', $item['complete_file'], $s3_url)) {
    $files[] = $item['complete_file'];
  }

Assuming you are looking for "https://s3.amazonaws.com/" within $item['storage_file'] and $item['complete_file'], I am submitting a patch that replaces preg_match with strstr:

  $s3_url = variable_get('mm_s3_default_server_url', "https://s3.amazonaws.com/");
  // build a list of files we are going to delete
  if (strstr($item['storage_file'], $s3_url)) {
    $files[] = $item['storage_file'];
  }
  if (strstr($item['complete_file'], $s3_url)) {
    $files[] = $item['complete_file'];
  } AttachmentSize mm_s3.bug34.diff309 bytes

To recreate the problem:
- create nodetype with a filefield and an emfield.
- create mediamover config that harvests files uploaded to this filefield, uploads them to youtube, and populates the emfield with the corresponding youtube-url's.
- create a node, and upload a file to this filefield.
- run the mediamover config

When the config is run, media_youtube adds entries to the database tables db.media_youtube_node_data and db.media_youtube_metadata, corresponding to the newly-generated youtube-url.

If the user now edits the node and deletes the file, then mm_emfield deletes the corresponding youtube-url. But the corresponding entries in the media_youtube database tables don't get deleted.

I am attaching a patch to mm_emfield_node_file_delete() that fixes the problem.

function mm_emfield_node_file_delete($configuration, $file) {
  ...
      if ($node_field['embed'] == $url) {
if ($node_field['provider'] == 'youtube' && function_exists('media_youtube_emfield_field_extra_delete')){
  // About to delete youtube-url.  Delete corresponding entries from media_youtube database tables
media_youtube_emfield_field_extra_delete($node_field['value'], $node->vid, $field, $key);
}
        ...
      }
  ....

The function media_youtube_emfield_field_extra_delete() deletes the corresponding entries from the media_youtube database tables. This function is defined in a separate patch that I submitted to the media_youtube queue: http://drupal.org/node/1049234

AttachmentSize mm_emfield.bug25.diff337 bytes

Hi,

I am working on Media mover module and facing 2-3 issues while uploading files to FTP server in following scenarios.

1. Status remains same after successfully uploaded file to FTP server in media_mover_files table
   1. I had uploaded small size file (33kb) and perfectly uploaded to FTP server. In database (- media_mover_files table), status of this file was 8 that is Finished (MMA_FILE_STATUS_COMPLETE_COMPLETE)
   3. Next, I tried to upload 11 MB file and it was uploaded on FTP server but the status of the file was not updated to 8 (MMA_FILE_STATUS_COMPLETE_COMPLETE), it was 5 (Storing) (MMA_FILE_STATUS_IN_STORAGE)
   5. Then, again I ran the cron and this time previously uploaded file(11 MB) started to upload. After completing the upload file status in table is 5
2. Cron is not uploading files which was not processed in last cron run
3. Large size file is partially uploaded on FTP server and its not being uploaded in next cron run

I am able to install media_mover-6.x-1.x-dev (dated 22 March 2012) by first installing 1.0-beta9, and then upgrading to 1.x-dev.
However, if I try to install 1.x-dev without first installing 1.0-beta9, then I get this error when I try to enable the media_mover_api module:

user warning: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'fid (cid, fid), INDEX cid (cid), INDEX harvest_file (harvest_file), INDEX pro' at line 16 query: CREATE TABLE media_mover_files ( `mmfid` INT auto_increment DEFAULT NULL, `nid` INT unsigned DEFAULT NULL, `fid` INT unsigned DEFAULT NULL, `cid` INT unsigned DEFAULT NULL, `harvest_file` VARCHAR(255) DEFAULT '', `process_file` VARCHAR(255) DEFAULT '', `storage_file` VARCHAR(255) DEFAULT '', `complete_file` VARCHAR(225) DEFAULT '', `status` INT unsigned DEFAULT NULL, `date` INT unsigned DEFAULT NULL, `data` MEDIUMTEXT DEFAULT NULL, PRIMARY KEY (mmfid), INDEX mmfid (mmfid), INDEX nid (nid), INDEX cid,fid (cid, fid), INDEX cid (cid), INDEX harvest_file (harvest_file), INDEX process_file (process_file) ) /*!40100 DEFAULT CHARACTER SET utf8 */ in /mysite/includes/database.inc on line 551.

Also, fyi, when you install 6.x-1.x-dev, the version listed in the .info file and displayed on the modules page is 6.x-1.0-beta7+24-dev

I have a configuration that harvests a particular filefield and stores the files on s3.
If the user deletes a node, the file entries do not get deleted from the 'files' database table.
The attached patch to mm_s3_delete_file() fixes the problem.

function mm_s3_delete_file($item, $configuration) {
   // Delete file entry in in 'files' table
  $fid = $item['fid'];
  db_query('DELETE FROM {files} WHERE fid = "%d"', $fid); // delete entry in 'files' table
  ...
} AttachmentSize mm_s3.bug19.diff184 bytes

When the user deletes a node, mm_emfield_media_mover() gets called with $op='delete', which then calls mm_emfield_node_file_delete(), which then calls node_save().

function mm_emfield_media_mover($op, $action = null, $configuration = null, &$file = array(), $job = null, $nid = null) {
  if (module_exists('emfield')) {
    switch ($op) {
      ...
      case 'delete':
        if ($action == MM_EMFIELD_STORE || $action == MM_EMFIELD_COMPLETE) {
          mm_emfield_node_file_delete($configuration, $file);
        }
        break;
    }
  }

As a result, node_save() gets called while node_delete() is in the midst of executing. This causes a bunch of problems when a node is deleted, including:

1. The emfield rows in content_field_myemfield, corresponding to the deleted node's harvested files, are not deleted. (The columns are instead set to NULL, because node_save() causes hook_nodeapi to get called with $op='update'.) See http://drupal.org/node/1497418
2. Similarly, the rows in the content_field_myfilefield database table, corresponding to the deleted node's harvested files, are not deleted.
3. Emfield outputs error message "unserialize() expects parameter 1 to be string". See http://drupal.org/node/1308856.
4. Multiple calls to hook_nodeapi with $op='update', when user deletes node.

I modified media_mover_api.module so that mm_emfield_node_file_delete() is no longer called when a node is deleted. As a result, node_save() is no longer called in the midst of executing node_delete(), and all of the problems listed above went away.

The basic idea was to add an $op to hook_media_mover(), to distinguish between "user has deleted a node" ($op='node_delete') and "user has deleted a file" ($op='delete'). mm_emfield's implementation of hook_media_mover is then able to distinguish these two cases, and call mm_emfield_node_file_delete() only when the user has deleted a file, but not when the user has deleted a node. In this way, node_save() is no longer called in the midst of executing node_delete().

Because this approach distinguishes 'node_delete' from 'delete', it may be necessary to modify the implementation of hook_media_mover() in some other modules. For example, I found it was necessary to add a case statement for $op='node_delete' to mm_s3's implementation of hook_media_mover(), so that mm_s3_delete_file() would get called both when the user deleted a node and when the user deleted a file.

Here are the detailed code changes:

function media_mover_api_node_files_delete()
In the call to media_mover_api_file_delete_call(), I pass $op='node_delete' as a second parameter, to indicate that the user had deleted a node.

function media_mover_api_node_files_delete($node) {
     ...
      if ($configuration->settings->mma_node_item_delete) {
        foreach ($mm_files as $mm_file) {
           // media_mover_api_file_delete_call($mm_file); // original code
           media_mover_api_file_delete_call($mm_file, 'node_delete'); // <--- HERE.  added second parameter
        }
      ...
}

function media_mover_api_file_delete_call()
The function accepts a second parameter, which by default is 'delete', and passes this parameter to _media_mover_api_module_invoke().

function media_mover_api_file_delete_call($file, $op='delete') {
  foreach (media_mover_api_verbs() as $verb) {
    // _media_mover_api_module_invoke($configuration->{$verb}->module, 'media_mover', 'delete', $configuration->{$verb}->action, $configuration->{$verb}->configuration, $file, $configuration); // original code
  _media_mover_api_module_invoke($configuration->{$verb}->module, 'media_mover', $op, $configuration->{$verb}->action, $configuration->{$verb}->configuration, $file, $configuration); // <--- HERE.  replaced 'delete' with $op

  }

With the changes above, mm_emfield_media_mover() no longer calls mm_emfield_node_file_delete() when a node has been deleted.

function mm_s3_media_mover()
I modified mm_s3's implementation of hook_media_mover to support these changes.

function mm_s3_media_mover($op, $action_id = null, $configuration = array(), &$file = array()) {
   switch ($op) {
     case 'node_delete':  // <--- HERE.  Added this line, so mm_s3_delete_file() gets called when node is deleted
     case 'delete':
        mm_s3_delete_file($file, $configuration);
     break;
   }
} AttachmentSize media_mover_api.bug20.diff674 bytes mm_s3.bug20.diff47 bytes

I have a configuration that uploads files to s3. Suppose I opt to "delete source from local server" after file is stored on s3.

If a second file is uploaded with same name ('abc.flv') as a previously uploaded file, then this second file does not get renamed to 'abc_0.flv' (because 'abc.flv' no longer resides on the local server), and media_mover does not process this second file.

To fix the problem, I modified mm_s3_send() so that rather than deleting the source-file, it replaces the source-file with a 0-byte file. This way, when the second file is uploaded, it gets renamed to 'abc_0.flv'.

function mm_s3_send($file, $configuration, $verb, $action_id) {
   ...
   foreach (array_unique($delete_files) as $delete_file) {
      watchdog('MM S3', 'Deleted local file: !file', array('!file' => $delete_file));
      unlink($delete_file);
      touch($delete_file); // <--- this is the line I added
    }
} AttachmentSize mm_s3.bug21.diff38 bytes

I need to implement ftp funxtionality on my web site were a user can upload a file using ftp. Is this module good for this purpose ?

Thanks
Tom

The function mm_cck_field_filefield() contains the following code:

  // we need to alter the file object slightly so it will display correctly
  $new_file['data'] = array('description' => '');

This sometimes causes an "unserialize" warning from function filefield_field_load(), when a node is deleted.

warning: unserialize() expects parameter 1 to be string, array given in /mysite/sites/all/modules/filefield/filefield_field.inc on line 139.

function filefield_field_load($node, $field, &$items, $teaser, $page) {
  if (empty($items)) {
    return array();
  }
  foreach ($items as $delta => $item) {
    // Despite hook_content_is_empty(), CCK still doesn't filter out
    // empty items from $op = 'load', so we need to do that ourselves.
    if (empty($item['fid']) || !($file = field_file_load($item['fid']))) {
      $items[$delta] = NULL;
    }
    else {
      if (isset($item['data']) && !empty($item['data'])) {
        $item['data'] = unserialize($item['data']); // <----- HERE is line 139
      }
     ...
}
$items['data'] is sometimes an array, rather than a string, and that is why "unserialize" fails. Specifically:
$item['data'] = array('description'=>'')

The "unserialize" warning goes away when I comment out the line of code from mm_cck.module that seems to be causing the problem, $new_file['data'] = array('description' => '');

I am submitting, for your consideration, a patch to mm_cck.module that deletes this line of code.

Thanks,
Mindy

AttachmentSize mm_cck.diff144 bytes

I want to ask about the YouTube support for media mover ??

I am seeing regular errors in my logs along the lines of

Parameter 4 to mm_cck_media_mover() expected to be a reference, value given in /var/www/drupal-6.25/includes/module.inc on line 476

I looked in the media_mover_api.module and found one instance where module_invoke() was *NOT* called (instead using call_user_func_array), and a comment immediately preceding that line saying that module_invoke doesn't seem to pass by reference correctly :

588: // @NOTE not using module_invoke here b/c it doesn't seem to pass the $file by reference correctly ($file)
589: $filepath = call_user_func_array($config->{$verb}->module .'_media_mover', array($verb, $config->{$verb}->action, $config->{$verb}->configuration, &$file, $config));

This seems to be an acknowledgement of the call by reference problem which I am seeing, which is why I am surprised to see several subsequent calls, with the file parameter, sent through module invoke

1155: module_invoke($file["{$verb}_module"], 'media_mover', 'update', $file["{$verb}_action"], NULL, $file);

2856: if ($new_items = module_invoke($configuration->{$verb}->module, 'media_mover', 'fetch', $item_config->{$verb}->action, $configuration, $item)) {

3014: module_invoke($configuration->{$verb}->module, 'media_mover', 'delete', $configuration->{$verb}->action, $configuration->{$verb}->configuration, $file, $configuration);

3092: module_invoke($configuration->{$verb}->module, 'media_mover', 'delete', $configuration->{$verb}->action, $configuration->{$verb}->configuration, $file, $configuration);

It seems that if module_invoke() is going to generate errors like I'm seeing, then it shouldn't be used anywhere that the 4th $file parameter is being passed. I can't decide whether these subsequent calls to module_invoke are in fact the source of my problem, and a legitimate problem in the code, or whether I am just rationalizing that the problem isn't mine :=)

Does someone know why module_invoke with the $file parameter was replaced in one situation, but not in the others? Doesn't that alone seem inconsistent, and worthy of some investigation?

I've got a quick question. I've got some pretty huge video files that I've already got uploaded to Amazon S3. I'm using the video module as well as the amazonS3 module and they work fine for uploading smaller videos to AS3. However, big files are a problem (>450mb or bigger). I'm trying to find a way to attach a file to my node but I've thus far not found a solution. Can this module be used for that purpose? I've tried filefield_sources, and that not the right tool for the job.

If anyone could help me out I'd appreciate it.

The mm_s3_requirements (in mm_s3.install.php) contains a check for curl which includes a test for !@dl('curl.so'). I do not understand how the @ works when put before the call to dl() or if that is relevant, but the call causes a server error for me.

The full line is:

if (extension_loaded('curl') && !@dl(PHP_SHLIB_SUFFIX == 'so' ? 'curl.so' : 'php_curl.dll')) { $has_curl = true; }

The test for extension_loaded('curl') works just fine, and everything else about the site & code works fine if that !@dl() requirements check is removed. For what it's worth, I have confirmed that PHP_SHLIB_SUFFIX is 'so' so the call to dl() is using 'curl.so' as the parameter.

Could I be missing something in this new server install which allows dl() to function?

I have a node-type that contains a multi-valued filefield.
I have a mediamover-config that harvests the files from this field and uploads them to youtube.

If the user creates a node with a single file, then the 'status' for this file, in the media_mover_files table, is 8 (MMA_FILE_STATUS_COMPLETE_COMPLETE).

However, if the user uploads multiple files, then the 'status' for these files is 7 (MMA_FILE_STATUS_IN_COMPLETE).

That is, for some reason I do not understand, it seems that some final step in the processing of these files is not completed.

As a result, media_mover_api_node_files_fetch($nid) fails to fetch these files, because by default it looks for files with status = 8.

As a temporary work-around, I replaced:
$media_mover = media_mover_api_node_files_fetch($nid);
with

$media_mover = media_mover_api_node_files_fetch($nid);
if (!$media_mover){
  $media_mover = media_mover_api_node_files_fetch($nid, MMA_FILE_STATUS_IN_COMPLETE); 
}

The function media_mover_api_node_update($node) is supposed to delete any media-mover files that the user has marked for deletion, and also delete entries for these files in the media_mover_files table. But it is not working. There are two problems. First, the function assumes that information about the media-mover files is stored in $node->media_mover. But $node->media_mover is generally not set. Second, the function assumes that $node->files stores information about whether a file has been marked for removal. But $node->files is generally not set if the files are stored in a filefield, as opposed to being attached to the node itself.

I believe the attached patch fixes these problems. It also fixes the problem described here, that when a media-mover file gets deleted, its listing in the 'files' table is not deleted: http://drupal.org/node/1477820

AttachmentSize media_mover_api.diff2.71 KB

I have a configuration that harvests files from a particular filefield, and then moves the file to s3.

I noticed that after the file is moved to s3, it is still listed in the 'files' database table, as though it were still a local file. If the user then deletes the file, it remains listed in the 'files' table.

Presumably, once the user has deleted the file it should no longer be listed in the 'files' table. And I would like to create a patch to fix the problem. But I am not sure where the problem lies:

Should mm_s3 delete the file-listing from the 'files' table, when the file is moved to s3? Is this where the bug lies?

Or should media_mover_api delete the file-listing from the 'files' table, when the user deletes the file? Is this where the bug lies?

Thanks for your advice.

Suppose I have 3 configurations that harvest a particular filefield.
Is there a way to chain these configurations, so that the second configuration runs after the first configuration successfully completes, and the third configuration runs after the second configuration successfully completes?

The motivation is described here: http://drupal.org/node/1469780

Basically, I have one configuration that uploads a filefield to youtube and stores the youtube url in an emfield. I have a second configuration that uploads this filefield to s3 and then deletes the local file. I want to run the second configuration after the first configuration completes, because otherwise the local file might get deleted before it is successfully uploaded to youtube.

I have a configuration ("youtube-config") that:
- harvest: harvest a particular filefield
- process: upload file to youtube
- complete: store the youtube url in an emfield

I have a second configuration ("s3-config) that:
- harvest: harvest this same filefield
- storage: store the file on s3

Both configurations work fine.

I then tried combining the two configurations ("combo-config"):
- harvest: harvest the filefield
- process: upload file to youtube
- storage: store file on s3 and delete local copy
- complete: store the youtube url in an emfield

With this combo-config, the "complete" step never runs. I get this watchdog message:

Harvested file is not readable, check permissions: http://www.youtube.com/watch?v=abcdefg

Should I be able to combine "upload to s3" with "upload to youtube" in the same configuration?

If I go back to two separate configurations, is there a way to ensure that the s3-storage step does not delete the local file before the youtube-process step finishes uploading the local file to youtube?

For example, is there a way to ensure this sequence of steps:
harvest: harvest this filefield
process: upload to youtube
complete: store youtube url in emfield
storage: once the "complete" step is complete, store the file on s3 server and then delete the local copy

Thanks very much

I am writing custom code to programmatically customize an exported configuration, and then programmatically import this customized configuration as a new configuration.

I have two questions about importing the new configuration

Question #1: How to handle the cid

The exported configuration contains code that sets the cid in a bunch of places, eg:

$configuration->cid = '4';
$configuration->process->cid =  '4';
$configuration->complete->configuration['cid'] =  '4';

I can't use the same cid, because I want to create a new configuration.

What is the right way to handle this? Should I programmatically query the database for the next available cid? Or is there a better approach?

Question #2: How to do the import

I found this code in mediamover_api_io.inc. Is this the right way to import the new configuration?

  $id = time();
  cache_set("media_mover_config_$id", $configuration, 'cache');
  drupal_set_message(t('Successfully imported Media Mover configuration: !name.', array('!name' => $configuration->name)));

Thanks very much for your guidance,
Mindy