Topic: Importing customer csv data in FA v2.3.6

Just starting with FA, and it looks perfect for what we need, except for one major issue.  We're a manufacturing company in Brazil, we make only a few products but have over 500 customers.  I saw the module for importing items but none for importing customer csv data.  I also couldn't find any relevant answers in the forums.  I really hope this isn't a deal-breaker... FA seems to have everything we need but I can't spend the time re-entering customer data.
Thanks for any help!
ps  I'm not a programmer, so any answers should be idiot-proof!

Re: Importing customer csv data in FA v2.3.6

There has been a customer csv import module, but I don't think it is up to date. Maybe someone in the audience can help you.

/Joe

Re: Importing customer csv data in FA v2.3.6

Thanks, Joe!
Is there any way to test if the older module would work in version 2.3.6?  If so, how does one install it? I see that you can only install a module from withing the program... or is there a workaround?

Re: Importing customer csv data in FA v2.3.6

Hi robinbuzios,

To get an updated csv-file with all tables, you can make an export of your DB with PhpMyadmin.
If you make first time, made them from the demo-company to see sample data in all sheets (Think you setup a demo-company). Or put in the demo one customer with all the related data, to find in the xls-file better the relations between all the tables.

Select there "all tables", "xls", and "Put fields names in the first row".

Modify each sheet you need in the xls-file with your data.
Save them as csv-file, and import the csv-file with PhpMyAdmin in your DB (the table where you import data should be empty).

In this way you could import all data in the new company, not only your customers.

Best regards,
ferre

5 (edited by apmuthu 10/06/2014 09:28:50 pm)

Re: Importing customer csv data in FA v2.3.6

Please pass on the old customer import from csv module so that we can make it up-to-date. It does not seem to be available in the extensions repo.

On adding a customer through the FA Add Customer form the following SQLs are generated:

INSERT INTO `2_crm_persons` VALUES 
('2', 'Justin Zhou Lu', 'Justin Lu', NULL, NULL, NULL, NULL, NULL, NULL, NULL, '', '0');

INSERT INTO `2_crm_contacts` VALUES 
  (3, '2', 'cust_branch', 'general', '2')
, (4, '2', 'customer', 'general', '2');

INSERT INTO `2_cust_branch` VALUES
 (2, '2', 'Justin Zhou Lu', 'Justin Lu', '', '1', '1', '', 'DEF', '1', '', '4510', '1200', '4500', '1', '0', '', '0', 'Male', '0');

INSERT INTO `2_debtors_master` VALUES
 ('2', 'Justin Zhou Lu', 'Justin Lu', NULL, 'PQ453625N', 'USD', '1', '0', '0', '1', '4', '0', '0', '0', 'Male', '0');

Attached is what I understand to be the ERD between these tables (they are not fully depicted in the FA v2.3 ERD pdf).

Joe / Janusz - please explain if these relationships are correct with the 2_crm_contacts.type field enforcing some sort of filter.

There is no field for gender and CitizenID Number of the person in FA as of now.....GST / VAT Number field can be used for Citizen ID Number for now whilst gender can be placed in the Notes field but these land up in the 2_cust_branch and 2_debtors_master tables!

The id field of the 2_crm_contacts table does not seem to be referenced anywhere and is just used for uniqueness for view/edit purposes.

The cust_branch.branch_ref, debtors_master.debtor_ref and the crm_persons.name (and not the crm_persons.ref) are in the select boxes.

Post's attachments

FA_Customer_ERD_23.png 319.5 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

6 (edited by apmuthu 10/08/2014 07:43:21 am)

Re: Importing customer csv data in FA v2.3.6

Here is a pure MySQL import of one customer sequence for FA v2.3.22:

SET @currency=(SELECT MID(`value`,1,3) AS currency FROM `2_sys_prefs` WHERE category='setup.company' AND `name`='curr_default');  # SGD
SET @debtorsact=(SELECT (`value`+0) AS debtors_act FROM `2_sys_prefs` WHERE category='glsetup.sales' AND `name`='debtors_act');   # 1200
SET @paymentdiscact=(SELECT (`value`+0) AS debtors_act FROM `2_sys_prefs` WHERE category='glsetup.sales' AND `name`='default_prompt_payment_act');  # 4500
SET @salesdiscact=(SELECT (`value`+0) AS debtors_act FROM `2_sys_prefs` WHERE category='glsetup.sales' AND `name`='default_sales_discount_act');    # 4510
SET @ref='Jack Brown';
SET @name='Jack David';
SET @name2='Brown';
SET @address='123 Timbuktoo Street, Johannesburg, South Africa';
SET @phone='94568745';
SET @email='jack@dbrown.com';
SET @CitizenID='H5463546T';
SET @Gender='Male';

INSERT INTO 2_crm_persons (ref, `name`, name2, address, phone, email, notes) VALUES (@ref, @name, @name2, @address, @phone, @email, CONCAT(@CitizenID,' ',@Gender));

SET @personid=LAST_INSERT_ID();

INSERT INTO `2_debtors_master` (debtor_no, `name`, debtor_ref, address, tax_id, curr_code, credit_status, payment_terms, credit_limit, notes) 
  SELECT 
    NULL AS debtor_no, 
    CONCAT(`name`,' ',name2) AS `name`, 
    ref AS debtor_ref,
    address, 
    '' AS tax_id, 
    @currency AS curr_code, 
    1 AS credit_status,
    4 AS payment_terms, 
    0 AS credit_limit, 
    '' AS notes 
  FROM 2_crm_persons 
  WHERE id = @personid;

SET @debtorno=LAST_INSERT_ID();

INSERT INTO `2_cust_branch` (branch_code, debtor_no, br_name, branch_ref, br_address, area, salesman, contact_name, default_location, 
                                tax_group_id, sales_account, sales_discount_account, receivables_account, payment_discount_account, br_post_address, notes) 
  SELECT 
    NULL AS branch_code,
    @debtorno AS debtor_no,
    `name` AS br_name,
    debtor_ref AS branch_ref,
    address AS br_address,
    1 AS area,
    1 AS salesman,
    '' AS contact_name,
    'DEF' AS default_location,
    1 AS tax_group_id,
    '' AS sales_account,
    @salesdiscact AS sales_discount_account,
    @debtorsact AS receivables_account,
    @paymentdiscact AS payment_discount_account,
    address AS br_post_address,
    notes
  FROM `2_debtors_master`
  WHERE debtor_no = @debtorno;

SET @brcode=LAST_INSERT_ID();

INSERT INTO `2_crm_contacts` (id, person_id, `type`, `action`, entity_id) VALUES 
  (NULL, @personid, 'cust_branch', 'general', @brcode)
, (NULL, @personid, 'customer', 'general', @debtorno);

ERD is in the Wiki and attached in this post as well.

Post's attachments

FA_Contacts_ERD_1.png 30.4 kb, file has never been downloaded. 

You don't have the permssions to download the attachments of this post.

Re: Importing customer csv data in FA v2.3.6

Hello, I am a newbie adopter of the very excellent FA ERP system, and I'm writing code that will connect our Prestashop carts with FA. I found the above SQL code to be very helpful in figuring out what goes where. The table key link diagram was also very helpful.

But I am confused about 2 fields: "ref" and "name", and I'm hoping to get some clarification.

When I create a new customer through the FA interface, it asks for "Customer Name" and "Customer Short Name". Since customer names are not necessarily unique, I was careful to make up a unique code for the short name by using the first letter of first name + last name + source_ID, for example:

Customer Name: John Doe
Short Name: JDoe2522

When I enter the above in the FA interface, it first inserts a new record in the "fl_crm_persons" table, where it puts the name (John Doe) into the "ref" column, and the short name (JDoe2522) into the "name" column.

ref=John Doe
name=JDoe2522

It then copies various fields (as per SQL above) from this first insert into the "fl_debtors_master", where the "debtor_ref" column receives the "ref" column (which is the full name).

I'm wondering if this might be wrong, because it means the debtor_ref field in the "fl_debtors_master" may not have a unique value. Am I missing something here?

I was also wondering if there is a similar batch of SQL code to show what happens when a new order is entered?

Regards, Adrian

Re: Importing customer csv data in FA v2.3.6

It appears that you are not using the latest code. The ref and name fields were actually swapped while populating the table and the code was corrected in a commit on 2014-10-09. It would be best to use the stable snapshot.

Also, you seem to be using some non standard table prefix naming convention since FA usually uses numeric prefixes with an underscore but this will not be an issue here.

Furthermore, if you are using the en_GB Chart of Accounts from the official pkg repo then you will do well to do a line by line compare of the fields with the sql/en_US-new.sql in the db schema since in atleast 2 instances there are field order issues. As the devs are busy with getting FA v2.4 out, these remain in limbo.

Re: Importing customer csv data in FA v2.3.6

Ah, thank you so much for the clarification, I was thinking I missed something! Is there any kind of rough estimate about when v2.4 will be coming out? I may wait since the structures are changing.

Cheers, Adrian

Re: Importing customer csv data in FA v2.3.6

Do not wait for v2.4 - it's been in the making for years now.

Just make the corrections to the tables for benefitting from speed and removing redundant indices and standardising field types and order of fields. There is just too much testing to be done in v2.4 before it can be relied upon - besides, it's schema will keep changing for a while.

All my fixes to the schema in v2.3 are now in my GitHub Repo.

Stating the country you are in (work with) in your profile, will help newbies contact you.

Re: Importing customer csv data in FA v2.3.6

Hi, thanks so much for your prompt and helpful reply. We are in Canada, using the standard Canadian English chart of accounts.

I won't wait in that case, as it is mostly done and our fiscal year started a few days ago smile

Cheers, Adrian

12 (edited by apmuthu 01/04/2015 08:16:05 am)

Re: Importing customer csv data in FA v2.3.6

The standard Canadian Chart of Accounts too suffers from the field order issue. Check the position of the 0_cust_branch.branch_ref and 0_debtors_master.debtor_ref fields in the sql/en_US-new.sql and yours.

Re: Importing customer csv data in FA v2.3.6

Hi, I'm just wondering now what I have to do to make this work properly. I presume I need the Canadian chart of accounts because of tax issues, etc.

I see a new file in the SQL folder called "en_CA-general.sql" which I presume was put there when I installed the Canadian Chart of Accounts via the interface.

My question is this: is that the only file I have to change to make it match the "en_US-new.sql" file in the stable snapshot? (which I also downloaded).

I did the install with Softaculous and the version of FA I have is 2.3.22, which looks like the most current version. It defaulted to the fl_ prefix for the tables when I did the install, and I usually let the default happen, if I don't know any better. Should I have changed it to 0_ like what I see in the stable snapshot version? Will this give me trouble later?

Sorry for all the questions, but if I don't get this right our whole system is in jeopardy.

Regards, Adrian

14 (edited by apmuthu 01/04/2015 07:58:08 am)

Re: Importing customer csv data in FA v2.3.6

It is generally okay to install applications via Scriptaculous / Simple Scripts / Fantastico, and other one-click installers for testing and evaluation purposes and for long unsupported projects (later correct it and update it manually).

After the release of FA v2.3.22, several fixes have been added and these will not reflect in such installs.

Modifications that consultants use for their customers are rarely put out for public view. For my part, I have placed every single mod in the FAMods folder in my GitHub repo that I think will be of some use to any end user. This, I hope will provide for exhaustive testing and feedback - very little has come my way so far and is indicative of the closeness (euphemism for selfishness and possible malicious intent) with which most developers would like to keep critical code to themselves. The real value of a project is it's community and the trust it emanates to grow it's userbase.

Several of my mods have yet to make it to the core and extension repos for various reasons and all of them have been listed with a detailed changelog.

The change to the said sql file will be valid for new company creation only. All existing companies must be manually corrected with appropriate SQL statements.

If your system has been in jeopardy hitherto, be happy that you did not know it to be worried about it at least till now and were able to enjoy the Christmas and New Year celebrations whilst some of us at FA were busy saving everyone hassles in it's wake wink .

Re: Importing customer csv data in FA v2.3.6

1. Overwrite your scriptaculous install with the latest stable snapshot and what mods you desire after taking in the credentials from the config.php and config_db.php files
2. Keep the default company installed with en_US-new.sql and let the default prefix be 0_.
3. Go thru the web install (newer config.php variables will also get in) with the credentials from the earlier config_db.php and config.php files.
4. Login as default company admin.
5. Install the en_CA Chart of Accounts and download and edit the sql file for Canada updating it with the end result of the changes done in the stable snapshot (and recommended from my GitHub repo only if wished and understood) hitherto and upload it.
6. Create a new company with the modified Canadian CoA
7. Logout of the default company
8. Login to the newly created Canadian CoA company.
9. Take a backup of the install and preserve for initial state.
10. Make changes in the Setup menu items to suit your specific install.
11. log out of the Canadian Company.
12. All Canadian company nfo are now in your company/1/ folder
13. Purge the js cache in the Canadian company (rm company/1/js/*.js)
14. Login to the Canadian Company and work along....

16 (edited by ahrtgnbn 01/04/2015 05:06:26 pm)

Re: Importing customer csv data in FA v2.3.6

Hi @apmuthu, thanks for your very comprehensive reply, it is very much appreciated as I can see that I was headed down the wrong path by making assumptions about the veracity of my Softaculous install.

I now have my Prestashop connector working nicely and I can import all inventory and clients from each website in a flash, so I'm going to start over with a fresh install using the stable snapshot link you provided. I'll take a look at the changes you made in your GitHub repo and see if I can understand what that's about.

I did go into the system last night and find where the fields got apparently switched around in customers.php, but I think starting over is the best way to go, in case there's something else awry. I'm just glad I spotted the problem, which triggered this forum discourse.

You're right, the value of the project is in the community around it; without your help I probably would not have succeeded with this very important aspect of our business. We've been looking for a while for an ERP that can handle a multi-divisional company with multiple currencies, inventory control for multiple warehouse locations, retail sales and manufacturing. There isn't much else out there if you want to use good old php/mysql.

Cheers and thanks again, Adrian

17 (edited by apmuthu 01/04/2015 05:48:03 pm)

Re: Importing customer csv data in FA v2.3.6

Nice to see someone put their nose to the grind. Someday all of this will be useful to you in guiding newbies get on the FA bandwagon.

The core in my GitHub Repo is a verbatim copy of the stable snapshot (sometimes it is a tad delayed while I take my time to vet each change). You can fork the repo on GitHub whilst submitting patches back as well.

The files in the FAMods folder are to be used to over write their counterparts in the core. Take care when using those in the company folder and only do so if it satisfies your needs - some files will need to be used along with others whilst some others may be used alone. I use all of them. If you use all the FAMods files to overwrite the core ones, make sure you place the shcmds.sh (perms 4755, owner root or equivalent) above your webroot. It is used for some extra functionality in the Setup => Backup and Restore page.

The files in the extensions folder in the GitHub repo are to be used to overwrite their counterparts after installing from the official pkg repo. They are changes that have yet to make it to the official repo open to the public ($ read consultancy $).

Re: Importing customer csv data in FA v2.3.6

Well, we are escaping from a commercial desktop accounting program that can't be modified to suit our needs, so it is worth it to spend the time. I think the end result will be awesome and will save us an enormous amount of time wasted posting data that is already there as data. In some cases we are are re-entering data 3 times, sheesh.

I think our cost benefit will be at least one salary that can be re-directed into production.

I took a look at your mods, and for now they are a bit beyond me, so I think I'll wait until I have time to really study what they do.

Am I safe by using the snapshot by itself, without any mods? In other words, are these critical corrections or just enhancements?

Tks, Adrian

Re: Importing customer csv data in FA v2.3.6

My FAMods are critical for me especially the db schema and the added indices and standardised field types and sizes and defaults.

Use a program like WinMerge to study differences between files and files in folders.

Use Excel (or any spreadsheet) to generate your SQL update / insert statements and then use a MySQL client like SQLyog to execute them.

Re: Importing customer csv data in FA v2.3.6

You may also partake of info in the CoA differences post.

Re: Importing customer csv data in FA v2.3.6

Hi @apmuthu, thanks for the tip to that CoA differences post. I'm now going through the process of updating the Canadian CoA sql file (sql/en_CA-general.sql).

I have re-installed FA now from the snapshot, that went very smoothly. Thankfully, I was able to use the same db because one thing Softaculous does very nicely is a series of automatic daily rotating backups on our server. It's a bit of a bear to replicate that in php.

So I have looked at your FA_mods area and I'm going to assume that they are all good, and I will download and over-write them into each respective folder, with a view to your caution above.

Thanks again for all the help, it is appreciated, and maybe this forum thread will help other newbies too.

Regards, Adrian

22 (edited by apmuthu 01/05/2015 04:53:34 am)

Re: Importing customer csv data in FA v2.3.6

Yes, this thread has now become the "FA Install Tips" topic.....

Do not depend on Softaculous or any hosting service's backup facility for critical data - solely. Yes, it is convenient to start with, but deciphering which backup to revert to and what it's actual state was is to get lost in the maze. In fact, an uninstall in Softaculous will simply wipe out the entire FA and it's database!

Whenever you want to backup, FA has a nice feature within it's user interface - Setup => Backup and Restore - where you can put in a comment as to it's snapshot contents and state - and in my mod, rename it as well - and download it to your machine in gzipped form. This however requires you to take a backup of the company/# folder separately to match the sql backed up.

Assuming all my mods are good is indeed a vote of confidence - thanks - please provide feedback.

Re: Importing customer csv data in FA v2.3.6

Hi @apmuthu, the new title for this thread is a good one! I spent quite a lot of time today on the Canadian CoA sql file, and it was not a trivial undertaking. I don't know how you would ever do that without a *great* tool like WinMerge, thanks for the tip on that. I made a list of changes that I did, and I will publish them here once I'm sure it is correct.

So I got as far as creating the second company with the Canadian CoA and it worked ok. At first it reported errors in the sql file, and the nice thing is that FA told me where to look. So I went back and reversed some of the changes.

But the problem now is that I can't log into company 2, only the default company 1. The setup doesn't ask for a separate username for company 2, and I put the same pwd in as company 1, but no go. I presume the admin has the same username  for both companies?

I'm seeing some references to this problem elsewhere, but I can't quite put my finger on a solution - do you have any idea about this? As far as I know my server has all the proper configs as I have dozens of db's and I haven't had this problem before.

I went to delete the cached files in company 2 as per your instructions, but there was nothing there.

Regards, Adrian

24 (edited by apmuthu 01/05/2015 10:23:07 am)

Re: Importing customer csv data in FA v2.3.6

1. Delete company/2 folder and reset your credentials in config_db.php so that the counter - $tb_pref_counter - is set back and the array element for the second company is removed.
2. Empty (not truncate or drop) the database used for the company 2.
3. It is best to use a separate database for each company
4. The database should already have been created, a user privileged for it and privileges flushed.
5. Now test your CoA sql that it has no create database and use database statements and conforms to the standard en_US-new.sql format. Manually executing it on the target database and then emptying it out on successful execution is one way to fish for errors.
6. Login to default company as admin
7. Create your second company all over again, logout and login to the second company and go ahead.

All companies by default have the same administrator username of "admin" on creation. The username can only be changed in the database - phpMyAdmin / SQLyog. Within the FA Web UI, another administrator roled user can be created and then logged in using that and then the original one deleted but I do not know what will happen if some stupid script was to assume that user id 1 should always be admin!

The company name given during company creation is the one in the dropdown list during login that determines which db and FA company will be used. This info is stored in the company's entry in config_db.php. For security reasons, there is a variable in config.php - $text_company_selection - that can be set to make the dropdown box to a text box for manual entry.

Re: Importing customer csv data in FA v2.3.6

Good Morning @apmuthu, thanks for that info, it solved the whole problem. I did not realize that the new company (#2) would have the username 'admin', I was using the username of the first default company. I was beating my brains out until well after midnight on this one smile

I use Navicat and PHPmyAdmin for db management, so I can do the things you suggest, if it turns out to be necessary. I had also figured out how to change the company table prefix in the config_db.php file in the system root folder. That was tricky as I was looking for it in the db and ended up writing a routine to search every field name of every table in the db, and check the value.

So I will proceed today with your mods (after full backup) and see if we can get ths puppy working.

Cheers, Adrian