Import products from CSV with FormIt

I adopted the example import script to write a custom snippet that’s used as a FormIt hook. The form contains a simple file upload field and I get the products to import correctly, BUT:

  • the sku is not imported &
  • (maybe related to above) the info display of the linked product in the TV seems messed up:

Here is my snippet:

// load Commerce
$path = $modx->getOption('commerce.core_path', null, MODX_CORE_PATH . 'components/commerce/') . 'model/commerce/';
$params = ['mode' => $modx->getOption('commerce.mode')];
/** @var Commerce|null $commerce */
$commerce = $modx->getService('commerce', 'Commerce', $path, $params);

if (!($commerce instanceof Commerce)) {
  $modx->log(modX::LOG_LEVEL_ERROR, 'Could not load Commerce service from {' . $path . '}.');
  return false;
}


// get data from uploaded file

if (!empty($_FILES['importFile']['name']) ) {
  // open submitted file
  $csvFile = fopen($_FILES['importFile']['tmp_name'], 'r');
  
  // write file data into array
  $csvData = array();
  
  while (!feof($csvFile) ) {
    $csvData[] = fgetcsv($csvFile, 1000, ';');
  }

  // get header row
  $header = array_shift($csvData);
  
  // match headers to data in array
  $data = array();
  
  foreach ($csvData as $row) {
    if (!empty($row)) {
      $data[] = array_combine($header, $row);
    }
  }

  // close file
  fclose($csvFile);
  
} else {
  
  $errorMsg = 'Error.';
  $modx->log(modX::LOG_LEVEL_ERROR, $errorMsg);
  $hook->addError('importFile', $errorMsg);
  
  return false;
}


// get tax group
// this assumes all products have the same tax group and delivery type which is configured as the default
$taxGroup = $modx->getObject('comTaxGroup', ['id' => (int)$modx->getOption('commerce.default_tax_group')]);

if (!($taxGroup instanceof comTaxGroup)) {
  $modx->log(modX::LOG_LEVEL_ERROR, 'Tax group not found - configure that first.');
  return false;
}

// get delivery type
$deliveryType = $modx->getObject('comDeliveryType', ['id' => (int)$modx->getOption('commerce.default_delivery_type')]);

if (!($deliveryType instanceof comDeliveryType)) {
  $modx->log(modX::LOG_LEVEL_ERROR, 'Delivery type not found - configure that first.');
  return false;
}

// to use a specific non-default currency, use $commerce->getCurrency('KEY');
$currency = $commerce->currency;

// define resource settings
$resourceParent = 9;
$resourceTemplate = 22;
$tvName = 'products';
$tvType = 'list';       // list or matrix

foreach ($data as $sourceRow) {
  /** @var comProduct $product */
  $product = $modx->newObject('comProduct');
  $product->fromArray($sourceRow); // This assumes the source data has the right keys

  // set delivery type and tax group
  $product->set('tax_group', $taxGroup->get('id'));
  $product->set('delivery_type', $deliveryType->get('id'));

  // set pricing - for detailed explanation see https://docs.modmore.com/en/Commerce/v1/Developer/Products/Pricing.html
  $pricing = $product->getPricing($currency);
  $pricing->setRegularPrice(new \modmore\Commerce\Pricing\Price($currency, $sourceRow['price']));

  // have a sale price? 
  if (isset($sourceRow['sale_price']) && $sourceRow['sale_price'] > 0) {
    $pricing->addPriceType(
      new \modmore\Commerce\Pricing\PriceType\Sale(
        new \modmore\Commerce\Pricing\Price(
          $currency,
          $sourceRow['sale_price']
        ),
        new DateTime($sourceRow['sale_price_from']), // when the sale started
        new DateTime($sourceRow['sale_price_until']) // when the sale ended
      )
    );
  }
  $product->savePricing($pricing);

  $output = "Created product {" . $product->get('sku') . "} with ID #{" . $product->get('id') . "}";
  $modx->log(modX::LOG_LEVEL_ERROR, $output);

  // use the processor to create the resource - this does all sort of processing that we don't want to rewrite
  /** @var modProcessorResponse $response */
  $response = $modx->runProcessor('resource/create', [
    'context_key' => 'web',
    'parent' => $resourceParent,
    'template' => $resourceTemplate,
    'pagetitle' => $sourceRow['name'],
    'alias' => $modx->filterPathSegment($sourceRow['name'] . '-' . rand(0,999999)), // can leave this out or set a specific alias
    'published' => true,
  ]);
  
  if ($response->isError()) {
    $errors = implode(', ', $response->getAllErrors());
    $modx->log(modX::LOG_LEVEL_ERROR, 'Could not create resource: ' . $errors);
  } else {
    $resourceId = $response->getObject()['id'];
    $resource = $modx->getObject('modResource', ['id' => $resourceId]);
    
    if (!($resource instanceof modResource)) {
      $modx->log(modX::LOG_LEVEL_ERROR, 'Could not load created resource with ID: ' . $resourceId);
    } else {
      if ($tvType === 'list') {
        $tvValue = $product->get('id'); // Multiple products per resource? Separate IDs with a comma.
      } else { // matrix
        $modx->log(modX::LOG_LEVEL_ERROR, 'Matrix portion of this example is not yet ready :(');
        $tvValue = '';
      }
      $resource->setTVValue($tvName, $tvValue);
    }
  }
}

return true;

And here my test products.csv file:

sku;barcode;name;description;price;pricing;stock;stock_infinite;weight;weight_unit;image;tax_group;delivery_type;target;removed;removed_on;removed_by
1000100101;4015700000000;Product A;Lorem ipsum A;20075;;3;0;;;;;;;;;
1000100201;4004390000000;Product B;Lorem ipsum B;15087;;5;0;;;;;;;;;
1000100301;4015700000000;Product C;Lorem ipsum C;51573;;;1;;;;;;;;;

Also I would like to add the functionality that existing products get updated (based on the sku). How can I achieve this?

What does your $data array look like? I’m not immediately spotting any issues in the code, but perhaps if you do a var_dump($data) that shows if there’s an issue in the parsing.

If that’s parsed and formatted correctly, the following change may help.

$product->fromArray($sourceRow); 

to

$product->fromArray($sourceRow, '', true); 

That third parameter indicates it should also set keys.

Also I would like to add the functionality that existing products get updated (based on the sku). How can I achieve this?

Replace:

$product = $modx->newObject('comProduct');

with:

$product = $modx->getObject('comProduct', [
    'sku' => $sourceRow['sku'], 
    'removed' => false,
]);
if (!$product) {
    $product = $modx->newObject('comProduct');
}

Do note Commerce does not currently enforce unique SKUs so it is possible to get duplicates and unexpected behavior.

You’ll also need to change the processor it runs from resource/create to resource/update. Perhaps conditionally based on if a resource with the same name/parent/template already exists?

$exists = $modx->getCount('modResource', [ 
    'parent' => $resourceParent,
    'template' => $resourceTemplate,
    'pagetitle' => $sourceRow['name'],
]);
$processor = $exists > 0 ? 'resource/update' : 'resource/create';
$response = $modx->runProcessor($processor, [
//....

I did a print_r($data,1); to show the array in the log (not sure how to output it through FormIt otherwise):

Array
(
    [0] => Array
        (
            [sku] => 1000100101
            [barcode] => 4015700000000
            [name] => Product A
            [description] => Lorem ipsum A
            [price] => 20075
            [pricing] => 
            [stock] => 3
            [stock_infinite] => 0
            [weight] => 
            [weight_unit] => 
            [image] => 
            [tax_group] => 
            [delivery_type] => 
            [target] => 
             => 
            [removed_on] => 
            [removed_by] => 
        )

    [1] => Array
        (
            [sku] => 1000100201
            [barcode] => 4004390000000
            [name] => Product B
            [description] => Lorem ipsum B
            [price] => 15087
            [pricing] => 
            [stock] => 5
            [stock_infinite] => 0
            [weight] => 
            [weight_unit] => 
            [image] => 
            [tax_group] => 
            [delivery_type] => 
            [target] => 
             => 
            [removed_on] => 
            [removed_by] => 
        )

    [2] => Array
        (
            [sku] => 1000100301
            [barcode] => 4015700000000
            [name] => Product C
            [description] => Lorem ipsum C
            [price] => 51573
            [pricing] => 
            [stock] => 
            [stock_infinite] => 1
            [weight] => 
            [weight_unit] => 
            [image] => 
            [tax_group] => 
            [delivery_type] => 
            [target] => 
             => 
            [removed_on] => 
            [removed_by] => 
        )

)

(the empty string actually contains removed but I guess the discourse markup removes that).

Will look into the update part soon, thank you!

I tried changing this, but without luck, still no sku in the database.

I also created a product through the manager and noticed that this way the price in the db is set to 0? The pricing column still holds the actual value but is it supposed to be this way?

Yes, the price column is legacy from earlier Commerce versions. It’s only filled for some of the products because the import script fills it.

I’m very confused why it’s not saving the SKU for you. Does it work when you add it like this after the $product->fromArray()?

  $product->set('sku', (string)$sourceRow['sku']);

And is there anything in the MODX error log perhaps?

Does this mean I should avoid using it to define the product prices? Or will it continue to work just like it does now?

That’s still a no. Nothing in the logs either…

I also just tried altering the SKUs so they started with letters (just because the first one looks really binary if that could have an impact of some sort) but still nothing.

Alright, so I did some more testing and it looks like whatever is the first key in the array can’t be retrieved, although it is clearly there :woozy_face:

I tested this by simply adding a random key in the csv:

random;sku;barcode;name;description;price;pricing;stock;stock_infinite;weight;weight_unit;image;tax_group;delivery_type;target;removed;removed_on;removed_by
100110110;100110110;4015700000000;Product A;Lorem ipsum A;20075;{"EUR":{"regular_price":20075,"price_types":[]}};3;0;3.000;kg;images/products/100110110.jpg;1;1;0;0;0;0

Output:

// logging print_r($sourceRow,1)

Array
(
    [random] => 100110110
    [sku] => 100110110
    [barcode] => 4015700000000
    [name] => Product A
    [description] => Lorem ipsum A
    [price] => 20075
    [pricing] => {"EUR":{"regular_price":20075,"price_types":[]}}
    [stock] => 3
    [stock_infinite] => 0
    [weight] => 3.000
    [weight_unit] => kg
    [image] => images/products/100110110.jpg
    [tax_group] => 1
    [delivery_type] => 1
    [target] => 0
    [removed ] => 0
    [removed_on] => 0
    [removed_by] => 0
)

// logging 'keyname: ' . $sourceRow['key']:

random:
sku: 100110110
barcode: 4015700000000
name: Product A
description: Lorem ipsum A
price: 20075
pricing: {"EUR":{"regular_price":20075,"price_types":[]}}
stock: 3
stock_infinite: 0
weight: 3.000
weight_unit: kg
image: images/products/100110110.jpg
tax_group: 1
delivery_type: 1
target: 0
removed: 0
removed_on: 0
removed_by: 0

Same happens if I delete the sku in the csv, then I can’t get the barcode value (as it’s now the first column I guess).

Does this help in some way to find a solution (or better the issue causing this)? Maybe it has something to do with the array pointer? But I thought this wouldn’t matter, as long as we address the key directly…

Honestly, I have no idea :joy: The data clearly seems to be there when dumping the $sourceRow inside the loop so I don’t see a reason for that to become unavailable suddenly.

A quick follow-up question here:

When do I have to save the $product (or really any) object after setting values and properties on it? I noticed that $product->setProperty('key', $value); did not do anything if I would not save afterwards which seems as intended to me at first, but then I was wondering, as in your example import script, I don’t see any saving done, although you’re using the set function a few times e.g. for setting the tax_group.

Is there any other function called, which also does saving or what am I missing here?

To also update on the original issue:

TL;DR: I found that csv files exported from Excel contain a BOM (Byte Order Mark) if specified as UTF-8 during the export. This means  is added to the beginning of the file, making my first key not just sku but sku which caused the mentioned issues.


The tricky bit was really identifying the problem, as those characters would only get displayed after being UTF-8 encoded. I had quite the journey figuring that one out, so I thought I would share it here as well.

Full origin story

At first I noticed that the problem only occured from csv files exported from Excel. If I simply created a csv from within VSCode everything worked fine.

So I ended up printing the input array from two test files (one from Excel, one from VSCode) to the logs and they looked completely identical. For some reason I still decided to place them in a diff-checker and this got me the first hint, that sku might not be identical to sku:

As I was exporting from Excel to csv spcified as UTF-8, I had another search if more people had this issue and I came across this thread where someone suggested trying to utf8_encode the file data.

So I did and now my print_r of the imported data was a bit more revealing:

    [0] => Array
        (
            [0] => sku     // note the extra characters
            [1] => parent_sku
            [2] => barcode

A search for those ominous characters now revealed that those are the BOM for UTF-8 files and with that info, I could finally search for a solution to remove them before processing my data.

To get rid of the BOM I adjusted my import script like so:

$filepath = $_FILES[$input]['tmp_name'];

// remove UTF-8 BOM    
$fileContent = file_get_contents($filepath); 
file_put_contents($filepath, str_replace("\xEF\xBB\xBF", '', $fileContent));

After calling $product->setProperty(), you do indeed need to save. Not necessarily after each property, but when you’re done adding things to a product.

Typically, $product->save().

In the example import it is actually saving the product through $product->savePricing($pricing) - that internally calls $product->save() so it’s not necessary to call it again, if that is the last thing that changed to the product. But it never hurts to call $product->save() explicitly; if there is nothing to save to the database that will not do anything.

1 Like