Hi All,
I'm cross-posting this thread because I feel it's relative not only to
MivaScript developers,
but to Merchant developers and users as well.
In summary, the MvLOCKFILE tag has some issues, which could be responsible
for occasional duplicate order numbers in Merchant.
The problem existed in Merchant 4.13 (we could see the db.mv code).
Whether or not this issue has been corrected in compiled versions is unknown.
-Brian
Burns Enterprises
On Tue, 22 Feb 2005 11:06:27 -0500, Brian(Burns Enterprises)
<[email protected]> wrote:
> Interesting. I missed this from the docs:
>
> "However, if the second process does not issue an <MvLOCKFILE> before
> attempting to access the file in question, there is nothing to prevent
> this access from taking place. For this reason, we refer to the action
> of <MvLOCKFILE> as a lock request, rather than a lock: it is a request
> to other Miva Script processes that they 'respect' the current
> process's request for exclusive access. Another way of looking at this
> is: <MvLOCKFILE> does not interact with any other Miva Script tag,
> only other <MvLOCKFILE>s."
>
> So, the server/OS/filesystem is not aware of the "lock".
>
> However, this does not explain the previous problem, where MvLOCKFILE was being
> used by both scripts.
>
> Merchant was still incorrect as of 4.13
>
> <MvFUNCTION NAME="Key_Generate" PARAMETERS="type" STANDARDOUTPUTLEVEL="">
> <MvASSIGN NAME="l.key" VALUE=0>
> <MvIF EXPR="{_DB_Open_Keys()}">
> <MvLOCKFILE FILE="{g.MerchantPath $ 'keys.dbf'}">
> <MvFIND NAME="Keys" VALUE="{l.type}" EXACT>
> <MvIF EXPR="{NOT Keys.d.EOF}">
> <MvASSIGN NAME="Keys.d.maxvalue" VALUE="{Keys.d.maxvalue + 1}">
> <MvUPDATE NAME="Keys">
> <MvASSIGN NAME="l.key" VALUE="{Keys.d.maxvalue}">
> </MvIF>
> </MvLOCKFILE>
> </MvIF>
> <MvFUNCTIONRETURN VALUE="{l.key}">
> </MvFUNCTION>
>
> The only way I've found to create keys for a unique index is:
>
> <MvFUNCTION NAME="Key_Generate" PARAMETERS="name" STANDARDOUTPUTLEVEL="">
> <MvCOMMENT>1/4/2004 -Burns</MvCOMMENT>
> <MvASSIGN NAME="l.dir" VALUE="{Data_Dir()}">
> <MvLOCKFILE FILE="{l.dir $ 'keys.dbf'}">
> <MvOPEN NAME="KeysDB" DATABASE="{l.dir $ 'keys.dbf'}"
> INDEXES="{l.dir $ 'keys.mvx'}">
> <MvIF EXPR="{NOT MvOPEN_Error}">
> <MvFIND NAME="KeysDB" VALUE="{l.name}" EXACT>
> <MvIF EXPR="{NOT KeysDB.d.eof}">
> <MvASSIGN NAME="l.key" VALUE="{KeysDB.d.value + 1}">
> <MvASSIGN NAME="KeysDB.d.value" VALUE="{l.key}">
> <MvUPDATE NAME="KeysDB">
> </MvIF>
> <MvCLOSE NAME="KeysDB">
> </MvIF>
> </MvLOCKFILE>
> <MvFUNCRETURN VALUE="{l.key}">
> </MvFUNCTION>
>
>
> On Mon, 21 Feb 2005 16:40:39 -0800, Larry Hiscock
> <[email protected]> wrote:
> > Script #2 should attempt to lock the file before it deletes it. Under
> > Unix/Linux at least, a file can be deleted from underneath a process that
> > has it open.
> >
> > Larry Hiscock
> > AngelicHost
> >
> > -----Original Message-----
> > From: [email protected] [mailto:[email protected]] On Behalf
> > Of Brian(Burns Enterprises)
> > Sent: Monday, February 21, 2005 5:09 AM
> > To: MvMarkus
> > Cc: [email protected]
> > Subject: [meu] Re: MvLOCKFILE (was: Adding fields to a database )
> >
> > Thanks for the reply Markus!
> > I certainly appreaciate your 'thinking out load' on this.
> > You make some very interesting points.
> >
> > One other interesting test I performed just recently.....
> >
> > Script #1:
> > - Lock file
> > - re-create file (using MvExport or MvCreate)
> > - loop until timeout
> >
> > Scipt #2:
> > - while #1 is in loop, try to delete the file
> >
> > Well, it deleted it. Leaving nothing but the .lck file in the directory,
> > which disappeared once the script timed out.
> >
> > Ran into this while looking at a rebuilding a database.
> > I assumed that if used MvLockfile that Script #2 would timeout waiting for
> > the lock to release.
> >
> > There definately does appear to be some error in design.
> > In fact, I believe these issues are why every Merchant store will eventually
> > will suffer from the "duplicate order_id" error we all deal with from time
> > to time.
> >
> > On Mon, 21 Feb 2005 11:57:12 +0100, MvMarkus <[email protected]> wrote:
> > > Dear Bill, Brian,
> > >
> > > (I post this back to the mailing list, maybe other users can look into
> > > it as
> > > well)
> > >
> > > I just tested the scripts last night (on a Windows 2003 server / Miva
> > > Mia
> > > (uncompiled) and EmpresaVM and uncompiled on FreeBSD (sorry I was too
> > > lazy to also test it there compiled ). The Windows server has 1
> > > processor, the FreeBSD 2.
> > >
> > > As expected I was able to reproduce the errors on both machines and
> > > versions. Interesting is that Mia and Empresa occasionally showed
> > > different results in some tests.
> > >
> > > Some thinking aloud:
> > >
> > > My uneducated guess is that this is a threading/timing and/or -as you
> > > suspected- a memory issue. One process (thread) should lock the memory
> > > access and update the table and the index. But instead MvLOCKFILE
> > > appears to only work on the filesystem or it fails to "inform" other
> > > threads/processes that the data in memory has been updated.
> > >
> > > Empresa is not a "true" server program, but merely a sophisticated
> > > cgi-application, meaning that it does not really control the dataflow
> > > of parallel processes and memory access. Instead the operating system
> > > controls it.
> > >
> > > Empresa reloads different instances of tables into different areas of
> > > the memory, which probably don't even know of each other because they
> > > are controlled by different threads. So when thread 1 locks its
> > > memoryblock (and the original file), this does -at least temporarily-
> > > not affect thread 2, since the processors are so fast that they switch
> > > to thread 2 and while 1 has not had the time to flush the data to the
> > disk.
> > >
> > > Should MvLOCKFILE indeed update the RAM, then there is also the
> > > possibility that the table sits fully in the processor cache (after
> > > all, the tables are small enough to fit into it), and MvLOCKFILE
> > > doesn't have the time to update the RAM and disk before the second
> > > thread kicks in. For this it would be interesting to see if the same
> > > phenomenon happens also with larger tables (a few megabytes or so).
> > >
> > > There is also the possibility that actually the disk cache is to blame.
> > > Earlier hard disk hardly had any, today all drives have at least 2MB -
> > > so more than enough to reload a cached table while the original (now
> > > updated) data is still written to the disk.
> > >
> > > It is also interesting to observe that when you put MvLOCKFILE ->
> > > before <- the wait() in test 1 and 2 (I didn't try the other ones),
> > > the test performs correctly, without errors. This could support the
> > theories about the timing.
> > >
> > > If any of this is indeed the case, this could be that the original
> > > design might be a few years old, where neither processors, RAM nor
> > > disk were even remotely as fast as they are today. If this hasn't been
> > > updated ever since, then timing issues like these are not entirely
> > surprising.
> > >
> > > I'm aftraid that this is not a bug, but a design error.
> > >
> > > This has obviously some very important implications, and I think I
> > > have seen this issue in real life applications, not only with
> > > adding/updating but also with packing. I don't know if they are related.
> > >
> > > I've always struggled with occasional index corruptions on some
> > > tables, and that very reason (without having had such a clear test at
> > > hand), I always add additional safeguards if possible, and also close
> > > usually tables after packing. But looking at it so clearly, these
> > > tests are very illuminating when you design applications that are
> > > supposed to run under really heavy load.
> > >
> > > Great work, Brian!
> > >
> > > Markus
> > >
> > > -----Original Message-----
> > > From: Bill Guindon [mailto:[email protected]]
> > > Sent: Sonntag, 20. Februar 2005 21:18
> > > To: MvMarkus
> > > Subject: Fwd: Adding fields to a database
> > >
> > > subject modified to dodge your filters (I hope)
> > >
> > > Just want to make sure you saw this, and I'd be interested on your
> > > take on it. Brian and I have been looking at this for quite a while.
> > >
> > > SUMMARY: MvLOCKFILE does not work unless you lock the file before you
> > > open it, and unlock it after you close it.
> > >
> > > It seems that the data is either not written to the cache, or the
> > > file, until the close happens, so any reads done in between are not
> > reliable.
> > >
> > > ---------- Forwarded message ----------
> > > From: Brian Burns <[email protected]>
> > > Date: Sat, 19 Feb 2005 21:30:36 -0500
> > > Subject: [meu] Adding fields to a database
> > > To: [email protected]
> > >
> > > I actually spent quite some time researching a bug in MvLOCKFILE.
> > > I've posted this before, but never received a response.
> > > <A HREF ="http://www.mvdevelopment.com/mvlists2/10/88999.html">http://www.mvdevelopment.com/mvlists2/10/88999.html</A>
> > >
> > > I've posted the test script I used here:
> > > <A HREF ="http://www.becoded.com/lockfile_test.txt">http://www.becoded.com/lockfile_test.txt</A>
> > >
> > > To my knowledge, the problem I encountered still exists.
> > >
> > > -Brian
> > > Burns Enterprises
> > >
> > > From: MvMarkus <[email protected]>
> > > Date: Mon, 14 Feb 2005 20:37:44 +0100
> > > Subject: RE: [meu] Adding fields to a database
> > > To: Patrick Locke <[email protected]>, Sandy Rozhon
> > > <[email protected]>, Miva Userlist <[email protected]>
> > >
> > > Unless you have an index on the serial column, a relatively secure and
> > > easy way to increment a record is to rely on the physical order of the
> > > records in the db. For this, you don't use an index.
> > >
> > > MvOpen DB -> without <- any indices
> > > MvLockfile
> > > MvGO db row=bottom
> > > MvASSIGN =l.lastID VALUE=d.serial_column
> > > MvSetIndex all indices
> > > MvASSIGN =d.serial_column Value=l.lastID+1
> > > MvASSIGN = all other columns....
> > > MvADD
> > > /MvLockfile
> > >
> > > One should not rely on the built-in locking mechanism of
> > > MvADD/MvUPDATE and instead lock the table manually with MvLOCKFILE,
> > > because the function (especially opening the indices) might be quite I/O
> > expensive (slow).
> > >
> > > Obviously, this technique only works as long as the table is not
> > > physically reorganized. In that case, it's better to use an index on
> > > d.serial_column and/or do a lookup to check if the value already
> > > exists before adding a new one.
> > >
> > > Markus
> > >
> > > -----Original Message-----
> > > From: [email protected] [mailto:[email protected]] On
> > > Behalf Of Patrick Locke
> > > Sent: Montag, 14. Februar 2005 20:31
> > > To: Sandy Rozhon; Miva Userlist
> > > Subject: Re: [meu] Adding fields to a database
> > >
> > > I am not familiar with the database you are building but if it is not
> > > updated by many people you can use alias.d.recno for this. I have
> > > some dbs that I am the only one that updates the information and i use
> > > this method all the time to update information. But with multiple
> > > people being able to update, it sometimes can cause problems.
> > >
> > > Patrick
> > > ----- Original Message -----
> > > From: "Sandy Rozhon" <[email protected]>
> > > To: "Miva Userlist" <[email protected]>
> > > Sent: Saturday, February 12, 2005 1:58 PM
> > > Subject: [meu] Adding fields to a database
> > >
> > > > I'm a newcomer to programming Miva databases. All my previous
> > > > experience was on flat files.
> > > >
> > > > So my first effort to create, add, and delete records has been
> > > > successful, but now I see that having some sort of serial number on
> > > > each record might make it easier to do editing and/or deleting of a
> > > > specific record. Is it possible to add a new field, or do I have to
> > > > go through the creation process all over again? If so, what happens
> > > > to the data I've already collected?
> > > >
> > > > Are there approved ways of doing this sort of thing?
> > > >
> > > > Sandy
> > >