Improve Generate Data Model Performance


(Jose C Gomez) #1

I understand that based on the sheer size of the data schema this may not be possible. But I’m hoping there’s a way to make RegenData Model in such a way that it can run a “Differential” such that only modified entities have to be regenerated and it won’t take so long.
Maybe having a separate DLL for the _UD tables instead of dumping it all in the 910100.dll. I will admit I’ve done zero research in this area just putting this out to see if anyone bites. This may be entity framework limitation / issue though @Bart_Elia?

This seems like a silly request since most people never or hardly ever touch this area. But for people like me I waste collective hours in this feature a week :frowning:

image


(Chris Conn) #2

Yaaaaaassss!


(Brandon Anderson) #3

Not to mention scheduling down time. Because we all make sure that no one is using the system when we regenerate right? :wink:


(Jose C Gomez) #4

Sure… sure cough
image


(Jason Hawthorne) #5

Just curious, how long does this usually take on average? I thought that it was really slow on our system but now it seems I am not alone.


(Brandon Anderson) #6

between 13 and 15 minutes for us.


(Jose C Gomez) #7

±10-15 minutes depending on a bunch of factors…


(Jason Hawthorne) #8

OK So I wasn’t crazy. I was wondering if it was the tables I was changing or what. Seemed crazy long.


(Jose C Gomez) #9

But I sometimes have to do this 3 - 4 times in a day specially if I don’t plan my fields out all in advanced… which happens sometimes when I’m working on big projects I realize I need a new field in the middle of the day once or twice… So collectively that’s an hour or so worth of work. “Lost” to staring at the progress bar.

We try to plan ahead and add them all at once but inevitably we forget stuff


(Jason Hawthorne) #10

Yep, that is exactly what was happening to me. Add 3 fields, regen, realized after some testing you need a 4th, do a regen.

I know its a different beast and the benefits of the new system, but the old E9 fields made it so quick.


(Jose C Gomez) #11

Well you could still do that… add the Char01-50 and Number01-50 etc but its much nicer to have nice names and labels and properties…


(Simon Hall) #12

Yes we have been doing this quite a bit lately with getting all of our customisations and tweeks happening. One of the other gripes I have with regen is the clunky UI.

Right mouse on the data model, opens the work in progress “please be patient” dialog an the generate Data Model form, then you have to click on generate. Wouldn’t it be better to just have generate DM form then when you click on generate it opens the work in progress dialogue?


(Bart Elia) #13

Backstory…

I added a new guy to the team a couple of yours ago who was very performance conscious and he swore he would solve it. He tore thru it all weekend on his frustration and reported back on the Monday.

@##$$(&@@ that’s a lot of data the Entity Framework needs and our db is HUUGE…

I think every person on the team has gone through the same frustration trying to knock down the perf on it. I wrote the first generator personally with the architect of EF back in the day. His original code took 90 minutes… A couple of others have improved the organization, stability and minor performance gains but none have been able to change the volume of data EF needs at design or run time.

We’ve looked at a few approaches and we are making another run at it working with MSFT on a few research topics atm but nothing to advertise or promise. It’s equally frustrating but the alternative back in the day is dropping you into SQL or fixed stored procs and the migration would have really been horrendous without LINQ.

That is a topic near and dear to my heart. I wish I had a better answer :confused:


(Simon Hall) #14

Not being that technical, but isn’t just a Delta approach really all that is required, or is that what is being done now and it just takes that long?

From a simplistic view on the sql side (at least) when we add a new column to a table it appears to rebuild or creates the UD version of the table and recreates the associated table view, what else does it do?

Likely there is also some relationship between the volume of existing data in the table you are adding the column to, and also if you were setting a default/initial value also.

Interested to know more how it works and happy to be told different if I’m just talking through my hat.


(Bart Elia) #15

I need to see if I have the numbers on where time is spent. It is an interesting idea to just delta the ‘ud columns’ instead of the whole thing. It would not be 100% accurate of course so if not working, always give the user a regen all option (Similar to today’s use).

An interesting thought I’ll add to my research list…


(Andris Skulte) #16

Can the “OK, I’m finished” be made to be on top of all windows? If I’ve clicked somewhere else during the regen, the box gets buried, and I think it’s still crunching (but has been finished for 10 minutes!)…

Not sure if Alt+Tab picks it up or not. I think I could only see it when looking at the task bar icon preview windows. Seems like an easy improvement.


(Bart Elia) #17

That sounds like a ticket - agreed its annoying.


(Pierre Hogue) #18

But if you click again on the icon of the taskbar, it should show up …

But yes I agree, I even move the original position of the window in order to make sure the Finished window does not show behind!!!


(Bart Elia) #19

FYI - You know there is a command line for DM Regen? I know our Ops folks use it exclusively as part of a PowerShell script to regen, recycle app pool, etc.


(Simon Hall) #20

I just learnt something and it’s only 8am ish. :slight_smile: