MvLOCKFILE (was: Adding fields to a database )

Dear Bill, Brian,

(I post this back to the mailing list, maybe other users can look into it as
well)

I just tested the scripts last night (on a Windows 2003 server / Miva Mia
(uncompiled) and EmpresaVM and uncompiled on FreeBSD (sorry I was too lazy
to also test it there compiled ). The Windows server has 1 processor, the
FreeBSD 2.

As expected I was able to reproduce the errors on both machines and
versions. Interesting is that Mia and Empresa occasionally showed different
results in some tests.

Some thinking aloud:

My uneducated guess is that this is a threading/timing and/or -as you
suspected- a memory issue. One process (thread) should lock the memory
access and update the table and the index. But instead MvLOCKFILE appears to
only work on the filesystem or it fails to "inform" other threads/processes
that the data in memory has been updated.

Empresa is not a "true" server program, but merely a sophisticated
cgi-application, meaning that it does not really control the dataflow of
parallel processes and memory access. Instead the operating system controls
it.


Empresa reloads different instances of tables into different areas of the
memory, which probably don't even know of each other because they are
controlled by different threads. So when thread 1 locks its memoryblock (and
the original file), this does -at least temporarily- not affect thread 2,
since the processors are so fast that they switch to thread 2 and while 1
has not had the time to flush the data to the disk.

Should MvLOCKFILE indeed update the RAM, then there is also the possibility
that the table sits fully in the processor cache (after all, the tables are
small enough to fit into it), and MvLOCKFILE doesn't have the time to update
the RAM and disk before the second thread kicks in. For this it would be
interesting to see if the same phenomenon happens also with larger tables (a
few megabytes or so).

There is also the possibility that actually the disk cache is to blame.
Earlier hard disk hardly had any, today all drives have at least 2MB - so
more than enough to reload a cached table while the original (now updated)
data is still written to the disk.


It is also interesting to observe that when you put MvLOCKFILE -> before <-
the wait() in test 1 and 2 (I didn't try the other ones), the test performs
correctly, without errors. This could support the theories about the timing.


If any of this is indeed the case, this could be that the original design
might be a few years old, where neither processors, RAM nor disk were even
remotely as fast as they are today. If this hasn't been updated ever since,
then timing issues like these are not entirely surprising.

I'm aftraid that this is not a bug, but a design error.


This has obviously some very important implications, and I think I have seen
this issue in real life applications, not only with adding/updating but also
with packing. I don't know if they are related.

I've always struggled with occasional index corruptions on some tables, and
that very reason (without having had such a clear test at hand), I always
add additional safeguards if possible, and also close usually tables after
packing. But looking at it so clearly, these tests are very illuminating
when you design applications that are supposed to run under really heavy
load.

Great work, Brian!

Markus






-----Original Message-----
From: Bill Guindon [mailto:[email protected]]
Sent: Sonntag, 20. Februar 2005 21:18
To: MvMarkus
Subject: Fwd: Adding fields to a database

subject modified to dodge your filters (I hope)

Just want to make sure you saw this, and I'd be interested on your take on
it. Brian and I have been looking at this for quite a while.

SUMMARY: MvLOCKFILE does not work unless you lock the file before you open
it, and unlock it after you close it.

It seems that the data is either not written to the cache, or the file,
until the close happens, so any reads done in between are not reliable.


---------- Forwarded message ----------
From: Brian Burns <[email protected]>
Date: Sat, 19 Feb 2005 21:30:36 -0500
Subject: [meu] Adding fields to a database
To: [email protected]


I actually spent quite some time researching a bug in MvLOCKFILE.
I've posted this before, but never received a response.
<A HREF ="http://www.mvdevelopment.com/mvlists2/10/88999.html">http://www.mvdevelopment.com/mvlists2/10/88999.html</A>

I've posted the test script I used here:
<A HREF ="http://www.becoded.com/lockfile_test.txt">http://www.becoded.com/lockfile_test.txt</A>

To my knowledge, the problem I encountered still exists.

-Brian
Burns Enterprises

From: MvMarkus <[email protected]>
Date: Mon, 14 Feb 2005 20:37:44 +0100
Subject: RE: [meu] Adding fields to a database
To: Patrick Locke <[email protected]>, Sandy Rozhon
<[email protected]>, Miva Userlist <[email protected]>

Unless you have an index on the serial column, a relatively secure and easy
way to increment a record is to rely on the physical order of the records in
the db. For this, you don't use an index.

MvOpen DB -> without <- any indices
MvLockfile
MvGO db row=bottom
MvASSIGN =l.lastID VALUE=d.serial_column
MvSetIndex all indices
MvASSIGN =d.serial_column Value=l.lastID+1
MvASSIGN = all other columns....
MvADD
/MvLockfile

One should not rely on the built-in locking mechanism of MvADD/MvUPDATE and
instead lock the table manually with MvLOCKFILE, because the function
(especially opening the indices) might be quite I/O expensive (slow).

Obviously, this technique only works as long as the table is not physically
reorganized. In that case, it's better to use an index on d.serial_column
and/or do a lookup to check if the value already exists before adding a new
one.

Markus

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Patrick Locke
Sent: Montag, 14. Februar 2005 20:31
To: Sandy Rozhon; Miva Userlist
Subject: Re: [meu] Adding fields to a database

I am not familiar with the database you are building but if it is not
updated by many people you can use alias.d.recno for this. I have some dbs
that I am the only one that updates the information and i use this method
all the time to update information. But with multiple people being able to
update, it sometimes can cause problems.

Patrick
----- Original Message -----
From: "Sandy Rozhon" <[email protected]>
To: "Miva Userlist" <[email protected]>
Sent: Saturday, February 12, 2005 1:58 PM
Subject: [meu] Adding fields to a database

> I'm a newcomer to programming Miva databases. All my previous
> experience was on flat files.
>
> So my first effort to create, add, and delete records has been
> successful, but now I see that having some sort of serial number on
> each record might make it easier to do editing and/or deleting of a
> specific record. Is it possible to add a new field, or do I have to
> go through the creation process all over again? If so, what happens
> to the data I've already collected?
>
> Are there approved ways of doing this sort of thing?
>
> Sandy

Re: MvLOCKFILE (was: Adding fields to a database )



Thanks for the reply Markus!
I certainly appreaciate your 'thinking out load' on this.
You make some very interesting points.

One other interesting test I performed just recently.....

Script #1:
- Lock file
- re-create file (using MvExport or MvCreate)
- loop until timeout

Scipt #2:
- while #1 is in loop, try to delete the file

Well, it deleted it. Leaving nothing but the .lck file in the directory,
which disappeared once the script timed out.

Ran into this while looking at a rebuilding a database.
I assumed that if used MvLockfile that Script #2 would timeout
waiting for the lock to release.

There definately does appear to be some error in design.
In fact, I believe these issues are why every Merchant store will eventually
will suffer from the "duplicate order_id" error we all deal with from
time to time.


On Mon, 21 Feb 2005 11:57:12 +0100, MvMarkus <[email protected]> wrote:
> Dear Bill, Brian,
>
> (I post this back to the mailing list, maybe other users can look into it as
> well)
>
> I just tested the scripts last night (on a Windows 2003 server / Miva Mia
> (uncompiled) and EmpresaVM and uncompiled on FreeBSD (sorry I was too lazy
> to also test it there compiled ). The Windows server has 1 processor, the
> FreeBSD 2.
>
> As expected I was able to reproduce the errors on both machines and
> versions. Interesting is that Mia and Empresa occasionally showed different
> results in some tests.
>
> Some thinking aloud:
>
> My uneducated guess is that this is a threading/timing and/or -as you
> suspected- a memory issue. One process (thread) should lock the memory
> access and update the table and the index. But instead MvLOCKFILE appears to
> only work on the filesystem or it fails to "inform" other threads/processes
> that the data in memory has been updated.
>
> Empresa is not a "true" server program, but merely a sophisticated
> cgi-application, meaning that it does not really control the dataflow of
> parallel processes and memory access. Instead the operating system controls
> it.
>
> Empresa reloads different instances of tables into different areas of the
> memory, which probably don't even know of each other because they are
> controlled by different threads. So when thread 1 locks its memoryblock (and
> the original file), this does -at least temporarily- not affect thread 2,
> since the processors are so fast that they switch to thread 2 and while 1
> has not had the time to flush the data to the disk.
>
> Should MvLOCKFILE indeed update the RAM, then there is also the possibility
> that the table sits fully in the processor cache (after all, the tables are
> small enough to fit into it), and MvLOCKFILE doesn't have the time to update
> the RAM and disk before the second thread kicks in. For this it would be
> interesting to see if the same phenomenon happens also with larger tables (a
> few megabytes or so).
>
> There is also the possibility that actually the disk cache is to blame.
> Earlier hard disk hardly had any, today all drives have at least 2MB - so
> more than enough to reload a cached table while the original (now updated)
> data is still written to the disk.
>
> It is also interesting to observe that when you put MvLOCKFILE -> before <-
> the wait() in test 1 and 2 (I didn't try the other ones), the test performs
> correctly, without errors. This could support the theories about the timing.
>
> If any of this is indeed the case, this could be that the original design
> might be a few years old, where neither processors, RAM nor disk were even
> remotely as fast as they are today. If this hasn't been updated ever since,
> then timing issues like these are not entirely surprising.
>
> I'm aftraid that this is not a bug, but a design error.
>
> This has obviously some very important implications, and I think I have seen
> this issue in real life applications, not only with adding/updating but also
> with packing. I don't know if they are related.
>
> I've always struggled with occasional index corruptions on some tables, and
> that very reason (without having had such a clear test at hand), I always
> add additional safeguards if possible, and also close usually tables after
> packing. But looking at it so clearly, these tests are very illuminating
> when you design applications that are supposed to run under really heavy
> load.
>
> Great work, Brian!
>
> Markus
>
> -----Original Message-----
> From: Bill Guindon [mailto:[email protected]]
> Sent: Sonntag, 20. Februar 2005 21:18
> To: MvMarkus
> Subject: Fwd: Adding fields to a database
>
> subject modified to dodge your filters (I hope)
>
> Just want to make sure you saw this, and I'd be interested on your take on
> it. Brian and I have been looking at this for quite a while.
>
> SUMMARY: MvLOCKFILE does not work unless you lock the file before you open
> it, and unlock it after you close it.
>
> It seems that the data is either not written to the cache, or the file,
> until the close happens, so any reads done in between are not reliable.
>
> ---------- Forwarded message ----------
> From: Brian Burns <[email protected]>
> Date: Sat, 19 Feb 2005 21:30:36 -0500
> Subject: [meu] Adding fields to a database
> To: [email protected]
>
> I actually spent quite some time researching a bug in MvLOCKFILE.
> I've posted this before, but never received a response.
> <A HREF ="http://www.mvdevelopment.com/mvlists2/10/88999.html">http://www.mvdevelopment.com/mvlists2/10/88999.html</A>
>
> I've posted the test script I used here:
> <A HREF ="http://www.becoded.com/lockfile_test.txt">http://www.becoded.com/lockfile_test.txt</A>
>
> To my knowledge, the problem I encountered still exists.
>
> -Brian
> Burns Enterprises
>
> From: MvMarkus <[email protected]>
> Date: Mon, 14 Feb 2005 20:37:44 +0100
> Subject: RE: [meu] Adding fields to a database
> To: Patrick Locke <[email protected]>, Sandy Rozhon
> <[email protected]>, Miva Userlist <[email protected]>
>
> Unless you have an index on the serial column, a relatively secure and easy
> way to increment a record is to rely on the physical order of the records in
> the db. For this, you don't use an index.
>
> MvOpen DB -> without <- any indices
> MvLockfile
> MvGO db row=bottom
> MvASSIGN =l.lastID VALUE=d.serial_column
> MvSetIndex all indices
> MvASSIGN =d.serial_column Value=l.lastID+1
> MvASSIGN = all other columns....
> MvADD
> /MvLockfile
>
> One should not rely on the built-in locking mechanism of MvADD/MvUPDATE and
> instead lock the table manually with MvLOCKFILE, because the function
> (especially opening the indices) might be quite I/O expensive (slow).
>
> Obviously, this technique only works as long as the table is not physically
> reorganized. In that case, it's better to use an index on d.serial_column
> and/or do a lookup to check if the value already exists before adding a new
> one.
>
> Markus
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf
> Of Patrick Locke
> Sent: Montag, 14. Februar 2005 20:31
> To: Sandy Rozhon; Miva Userlist
> Subject: Re: [meu] Adding fields to a database
>
> I am not familiar with the database you are building but if it is not
> updated by many people you can use alias.d.recno for this. I have some dbs
> that I am the only one that updates the information and i use this method
> all the time to update information. But with multiple people being able to
> update, it sometimes can cause problems.
>
> Patrick
> ----- Original Message -----
> From: "Sandy Rozhon" <[email protected]>
> To: "Miva Userlist" <[email protected]>
> Sent: Saturday, February 12, 2005 1:58 PM
> Subject: [meu] Adding fields to a database
>
> > I'm a newcomer to programming Miva databases. All my previous
> > experience was on flat files.
> >
> > So my first effort to create, add, and delete records has been
> > successful, but now I see that having some sort of serial number on
> > each record might make it easier to do editing and/or deleting of a
> > specific record. Is it possible to add a new field, or do I have to
> > go through the creation process all over again? If so, what happens
> > to the data I've already collected?
> >
> > Are there approved ways of doing this sort of thing?
> >
> > Sandy
>

Re: MvLOCKFILE (was: Adding fields to a database )



Script #2 should attempt to lock the file before it deletes it. Under
Unix/Linux at least, a file can be deleted from underneath a process that
has it open.

Larry Hiscock
AngelicHost


-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Brian(Burns Enterprises)
Sent: Monday, February 21, 2005 5:09 AM
To: MvMarkus
Cc: [email protected]
Subject: [meu] Re: MvLOCKFILE (was: Adding fields to a database )

Thanks for the reply Markus!
I certainly appreaciate your 'thinking out load' on this.
You make some very interesting points.

One other interesting test I performed just recently.....

Script #1:
- Lock file
- re-create file (using MvExport or MvCreate)
- loop until timeout

Scipt #2:
- while #1 is in loop, try to delete the file

Well, it deleted it. Leaving nothing but the .lck file in the directory,
which disappeared once the script timed out.

Ran into this while looking at a rebuilding a database.
I assumed that if used MvLockfile that Script #2 would timeout waiting for
the lock to release.

There definately does appear to be some error in design.
In fact, I believe these issues are why every Merchant store will eventually
will suffer from the "duplicate order_id" error we all deal with from time
to time.


On Mon, 21 Feb 2005 11:57:12 +0100, MvMarkus <[email protected]> wrote:
> Dear Bill, Brian,
>
> (I post this back to the mailing list, maybe other users can look into
> it as
> well)
>
> I just tested the scripts last night (on a Windows 2003 server / Miva
> Mia
> (uncompiled) and EmpresaVM and uncompiled on FreeBSD (sorry I was too
> lazy to also test it there compiled ). The Windows server has 1
> processor, the FreeBSD 2.
>
> As expected I was able to reproduce the errors on both machines and
> versions. Interesting is that Mia and Empresa occasionally showed
> different results in some tests.
>
> Some thinking aloud:
>
> My uneducated guess is that this is a threading/timing and/or -as you
> suspected- a memory issue. One process (thread) should lock the memory
> access and update the table and the index. But instead MvLOCKFILE
> appears to only work on the filesystem or it fails to "inform" other
> threads/processes that the data in memory has been updated.
>
> Empresa is not a "true" server program, but merely a sophisticated
> cgi-application, meaning that it does not really control the dataflow
> of parallel processes and memory access. Instead the operating system
> controls it.
>
> Empresa reloads different instances of tables into different areas of
> the memory, which probably don't even know of each other because they
> are controlled by different threads. So when thread 1 locks its
> memoryblock (and the original file), this does -at least temporarily-
> not affect thread 2, since the processors are so fast that they switch
> to thread 2 and while 1 has not had the time to flush the data to the
disk.
>
> Should MvLOCKFILE indeed update the RAM, then there is also the
> possibility that the table sits fully in the processor cache (after
> all, the tables are small enough to fit into it), and MvLOCKFILE
> doesn't have the time to update the RAM and disk before the second
> thread kicks in. For this it would be interesting to see if the same
> phenomenon happens also with larger tables (a few megabytes or so).
>
> There is also the possibility that actually the disk cache is to blame.
> Earlier hard disk hardly had any, today all drives have at least 2MB -
> so more than enough to reload a cached table while the original (now
> updated) data is still written to the disk.
>
> It is also interesting to observe that when you put MvLOCKFILE ->
> before <- the wait() in test 1 and 2 (I didn't try the other ones),
> the test performs correctly, without errors. This could support the
theories about the timing.
>
> If any of this is indeed the case, this could be that the original
> design might be a few years old, where neither processors, RAM nor
> disk were even remotely as fast as they are today. If this hasn't been
> updated ever since, then timing issues like these are not entirely
surprising.
>
> I'm aftraid that this is not a bug, but a design error.
>
> This has obviously some very important implications, and I think I
> have seen this issue in real life applications, not only with
> adding/updating but also with packing. I don't know if they are related.
>
> I've always struggled with occasional index corruptions on some
> tables, and that very reason (without having had such a clear test at
> hand), I always add additional safeguards if possible, and also close
> usually tables after packing. But looking at it so clearly, these
> tests are very illuminating when you design applications that are
> supposed to run under really heavy load.
>
> Great work, Brian!
>
> Markus
>
> -----Original Message-----
> From: Bill Guindon [mailto:[email protected]]
> Sent: Sonntag, 20. Februar 2005 21:18
> To: MvMarkus
> Subject: Fwd: Adding fields to a database
>
> subject modified to dodge your filters (I hope)
>
> Just want to make sure you saw this, and I'd be interested on your
> take on it. Brian and I have been looking at this for quite a while.
>
> SUMMARY: MvLOCKFILE does not work unless you lock the file before you
> open it, and unlock it after you close it.
>
> It seems that the data is either not written to the cache, or the
> file, until the close happens, so any reads done in between are not
reliable.
>
> ---------- Forwarded message ----------
> From: Brian Burns <[email protected]>
> Date: Sat, 19 Feb 2005 21:30:36 -0500
> Subject: [meu] Adding fields to a database
> To: [email protected]
>
> I actually spent quite some time researching a bug in MvLOCKFILE.
> I've posted this before, but never received a response.
> <A HREF ="http://www.mvdevelopment.com/mvlists2/10/88999.html">http://www.mvdevelopment.com/mvlists2/10/88999.html</A>
>
> I've posted the test script I used here:
> <A HREF ="http://www.becoded.com/lockfile_test.txt">http://www.becoded.com/lockfile_test.txt</A>
>
> To my knowledge, the problem I encountered still exists.
>
> -Brian
> Burns Enterprises
>
> From: MvMarkus <[email protected]>
> Date: Mon, 14 Feb 2005 20:37:44 +0100
> Subject: RE: [meu] Adding fields to a database
> To: Patrick Locke <[email protected]>, Sandy Rozhon
> <[email protected]>, Miva Userlist <[email protected]>
>
> Unless you have an index on the serial column, a relatively secure and
> easy way to increment a record is to rely on the physical order of the
> records in the db. For this, you don't use an index.
>
> MvOpen DB -> without <- any indices
> MvLockfile
> MvGO db row=bottom
> MvASSIGN =l.lastID VALUE=d.serial_column
> MvSetIndex all indices
> MvASSIGN =d.serial_column Value=l.lastID+1
> MvASSIGN = all other columns....
> MvADD
> /MvLockfile
>
> One should not rely on the built-in locking mechanism of
> MvADD/MvUPDATE and instead lock the table manually with MvLOCKFILE,
> because the function (especially opening the indices) might be quite I/O
expensive (slow).
>
> Obviously, this technique only works as long as the table is not
> physically reorganized. In that case, it's better to use an index on
> d.serial_column and/or do a lookup to check if the value already
> exists before adding a new one.
>
> Markus
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On
> Behalf Of Patrick Locke
> Sent: Montag, 14. Februar 2005 20:31
> To: Sandy Rozhon; Miva Userlist
> Subject: Re: [meu] Adding fields to a database
>
> I am not familiar with the database you are building but if it is not
> updated by many people you can use alias.d.recno for this. I have
> some dbs that I am the only one that updates the information and i use
> this method all the time to update information. But with multiple
> people being able to update, it sometimes can cause problems.
>
> Patrick
> ----- Original Message -----
> From: "Sandy Rozhon" <[email protected]>
> To: "Miva Userlist" <[email protected]>
> Sent: Saturday, February 12, 2005 1:58 PM
> Subject: [meu] Adding fields to a database
>
> > I'm a newcomer to programming Miva databases. All my previous
> > experience was on flat files.
> >
> > So my first effort to create, add, and delete records has been
> > successful, but now I see that having some sort of serial number on
> > each record might make it easier to do editing and/or deleting of a
> > specific record. Is it possible to add a new field, or do I have to
> > go through the creation process all over again? If so, what happens
> > to the data I've already collected?
> >
> > Are there approved ways of doing this sort of thing?
> >
> > Sandy
>

Re: MvLOCKFILE (was: Adding fields to a database )



Interesting. I missed this from the docs:

"However, if the second process does not issue an <MvLOCKFILE> before
attempting to access the file in question, there is nothing to prevent
this access from taking place. For this reason, we refer to the action
of <MvLOCKFILE> as a lock request, rather than a lock: it is a request
to other Miva Script processes that they 'respect' the current
process's request for exclusive access. Another way of looking at this
is: <MvLOCKFILE> does not interact with any other Miva Script tag,
only other <MvLOCKFILE>s."

So, the server/OS/filesystem is not aware of the "lock".

However, this does not explain the previous problem, where MvLOCKFILE was being
used by both scripts.

Merchant was still incorrect as of 4.13

<MvFUNCTION NAME="Key_Generate" PARAMETERS="type" STANDARDOUTPUTLEVEL="">
<MvASSIGN NAME="l.key" VALUE=0>
<MvIF EXPR="{_DB_Open_Keys()}">
<MvLOCKFILE FILE="{g.MerchantPath $ 'keys.dbf'}">
<MvFIND NAME="Keys" VALUE="{l.type}" EXACT>
<MvIF EXPR="{NOT Keys.d.EOF}">
<MvASSIGN NAME="Keys.d.maxvalue" VALUE="{Keys.d.maxvalue + 1}">
<MvUPDATE NAME="Keys">
<MvASSIGN NAME="l.key" VALUE="{Keys.d.maxvalue}">
</MvIF>
</MvLOCKFILE>
</MvIF>
<MvFUNCTIONRETURN VALUE="{l.key}">
</MvFUNCTION>


The only way I've found to create keys for a unique index is:

<MvFUNCTION NAME="Key_Generate" PARAMETERS="name" STANDARDOUTPUTLEVEL="">
<MvCOMMENT>1/4/2004 -Burns</MvCOMMENT>
<MvASSIGN NAME="l.dir" VALUE="{Data_Dir()}">
<MvLOCKFILE FILE="{l.dir $ 'keys.dbf'}">
<MvOPEN NAME="KeysDB" DATABASE="{l.dir $ 'keys.dbf'}"
INDEXES="{l.dir $ 'keys.mvx'}">
<MvIF EXPR="{NOT MvOPEN_Error}">
<MvFIND NAME="KeysDB" VALUE="{l.name}" EXACT>
<MvIF EXPR="{NOT KeysDB.d.eof}">
<MvASSIGN NAME="l.key" VALUE="{KeysDB.d.value + 1}">
<MvASSIGN NAME="KeysDB.d.value" VALUE="{l.key}">
<MvUPDATE NAME="KeysDB">
</MvIF>
<MvCLOSE NAME="KeysDB">
</MvIF>
</MvLOCKFILE>
<MvFUNCRETURN VALUE="{l.key}">
</MvFUNCTION>



On Mon, 21 Feb 2005 16:40:39 -0800, Larry Hiscock
<[email protected]> wrote:
> Script #2 should attempt to lock the file before it deletes it. Under
> Unix/Linux at least, a file can be deleted from underneath a process that
> has it open.
>
> Larry Hiscock
> AngelicHost
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf
> Of Brian(Burns Enterprises)
> Sent: Monday, February 21, 2005 5:09 AM
> To: MvMarkus
> Cc: [email protected]
> Subject: [meu] Re: MvLOCKFILE (was: Adding fields to a database )
>
> Thanks for the reply Markus!
> I certainly appreaciate your 'thinking out load' on this.
> You make some very interesting points.
>
> One other interesting test I performed just recently.....
>
> Script #1:
> - Lock file
> - re-create file (using MvExport or MvCreate)
> - loop until timeout
>
> Scipt #2:
> - while #1 is in loop, try to delete the file
>
> Well, it deleted it. Leaving nothing but the .lck file in the directory,
> which disappeared once the script timed out.
>
> Ran into this while looking at a rebuilding a database.
> I assumed that if used MvLockfile that Script #2 would timeout waiting for
> the lock to release.
>
> There definately does appear to be some error in design.
> In fact, I believe these issues are why every Merchant store will eventually
> will suffer from the "duplicate order_id" error we all deal with from time
> to time.
>
> On Mon, 21 Feb 2005 11:57:12 +0100, MvMarkus <[email protected]> wrote:
> > Dear Bill, Brian,
> >
> > (I post this back to the mailing list, maybe other users can look into
> > it as
> > well)
> >
> > I just tested the scripts last night (on a Windows 2003 server / Miva
> > Mia
> > (uncompiled) and EmpresaVM and uncompiled on FreeBSD (sorry I was too
> > lazy to also test it there compiled ). The Windows server has 1
> > processor, the FreeBSD 2.
> >
> > As expected I was able to reproduce the errors on both machines and
> > versions. Interesting is that Mia and Empresa occasionally showed
> > different results in some tests.
> >
> > Some thinking aloud:
> >
> > My uneducated guess is that this is a threading/timing and/or -as you
> > suspected- a memory issue. One process (thread) should lock the memory
> > access and update the table and the index. But instead MvLOCKFILE
> > appears to only work on the filesystem or it fails to "inform" other
> > threads/processes that the data in memory has been updated.
> >
> > Empresa is not a "true" server program, but merely a sophisticated
> > cgi-application, meaning that it does not really control the dataflow
> > of parallel processes and memory access. Instead the operating system
> > controls it.
> >
> > Empresa reloads different instances of tables into different areas of
> > the memory, which probably don't even know of each other because they
> > are controlled by different threads. So when thread 1 locks its
> > memoryblock (and the original file), this does -at least temporarily-
> > not affect thread 2, since the processors are so fast that they switch
> > to thread 2 and while 1 has not had the time to flush the data to the
> disk.
> >
> > Should MvLOCKFILE indeed update the RAM, then there is also the
> > possibility that the table sits fully in the processor cache (after
> > all, the tables are small enough to fit into it), and MvLOCKFILE
> > doesn't have the time to update the RAM and disk before the second
> > thread kicks in. For this it would be interesting to see if the same
> > phenomenon happens also with larger tables (a few megabytes or so).
> >
> > There is also the possibility that actually the disk cache is to blame.
> > Earlier hard disk hardly had any, today all drives have at least 2MB -
> > so more than enough to reload a cached table while the original (now
> > updated) data is still written to the disk.
> >
> > It is also interesting to observe that when you put MvLOCKFILE ->
> > before <- the wait() in test 1 and 2 (I didn't try the other ones),
> > the test performs correctly, without errors. This could support the
> theories about the timing.
> >
> > If any of this is indeed the case, this could be that the original
> > design might be a few years old, where neither processors, RAM nor
> > disk were even remotely as fast as they are today. If this hasn't been
> > updated ever since, then timing issues like these are not entirely
> surprising.
> >
> > I'm aftraid that this is not a bug, but a design error.
> >
> > This has obviously some very important implications, and I think I
> > have seen this issue in real life applications, not only with
> > adding/updating but also with packing. I don't know if they are related.
> >
> > I've always struggled with occasional index corruptions on some
> > tables, and that very reason (without having had such a clear test at
> > hand), I always add additional safeguards if possible, and also close
> > usually tables after packing. But looking at it so clearly, these
> > tests are very illuminating when you design applications that are
> > supposed to run under really heavy load.
> >
> > Great work, Brian!
> >
> > Markus
> >
> > -----Original Message-----
> > From: Bill Guindon [mailto:[email protected]]
> > Sent: Sonntag, 20. Februar 2005 21:18
> > To: MvMarkus
> > Subject: Fwd: Adding fields to a database
> >
> > subject modified to dodge your filters (I hope)
> >
> > Just want to make sure you saw this, and I'd be interested on your
> > take on it. Brian and I have been looking at this for quite a while.
> >
> > SUMMARY: MvLOCKFILE does not work unless you lock the file before you
> > open it, and unlock it after you close it.
> >
> > It seems that the data is either not written to the cache, or the
> > file, until the close happens, so any reads done in between are not
> reliable.
> >
> > ---------- Forwarded message ----------
> > From: Brian Burns <[email protected]>
> > Date: Sat, 19 Feb 2005 21:30:36 -0500
> > Subject: [meu] Adding fields to a database
> > To: [email protected]
> >
> > I actually spent quite some time researching a bug in MvLOCKFILE.
> > I've posted this before, but never received a response.
> > <A HREF ="http://www.mvdevelopment.com/mvlists2/10/88999.html">http://www.mvdevelopment.com/mvlists2/10/88999.html</A>
> >
> > I've posted the test script I used here:
> > <A HREF ="http://www.becoded.com/lockfile_test.txt">http://www.becoded.com/lockfile_test.txt</A>
> >
> > To my knowledge, the problem I encountered still exists.
> >
> > -Brian
> > Burns Enterprises
> >
> > From: MvMarkus <[email protected]>
> > Date: Mon, 14 Feb 2005 20:37:44 +0100
> > Subject: RE: [meu] Adding fields to a database
> > To: Patrick Locke <[email protected]>, Sandy Rozhon
> > <[email protected]>, Miva Userlist <[email protected]>
> >
> > Unless you have an index on the serial column, a relatively secure and
> > easy way to increment a record is to rely on the physical order of the
> > records in the db. For this, you don't use an index.
> >
> > MvOpen DB -> without <- any indices
> > MvLockfile
> > MvGO db row=bottom
> > MvASSIGN =l.lastID VALUE=d.serial_column
> > MvSetIndex all indices
> > MvASSIGN =d.serial_column Value=l.lastID+1
> > MvASSIGN = all other columns....
> > MvADD
> > /MvLockfile
> >
> > One should not rely on the built-in locking mechanism of
> > MvADD/MvUPDATE and instead lock the table manually with MvLOCKFILE,
> > because the function (especially opening the indices) might be quite I/O
> expensive (slow).
> >
> > Obviously, this technique only works as long as the table is not
> > physically reorganized. In that case, it's better to use an index on
> > d.serial_column and/or do a lookup to check if the value already
> > exists before adding a new one.
> >
> > Markus
> >
> > -----Original Message-----
> > From: [email protected] [mailto:[email protected]] On
> > Behalf Of Patrick Locke
> > Sent: Montag, 14. Februar 2005 20:31
> > To: Sandy Rozhon; Miva Userlist
> > Subject: Re: [meu] Adding fields to a database
> >
> > I am not familiar with the database you are building but if it is not
> > updated by many people you can use alias.d.recno for this. I have
> > some dbs that I am the only one that updates the information and i use
> > this method all the time to update information. But with multiple
> > people being able to update, it sometimes can cause problems.
> >
> > Patrick
> > ----- Original Message -----
> > From: "Sandy Rozhon" <[email protected]>
> > To: "Miva Userlist" <[email protected]>
> > Sent: Saturday, February 12, 2005 1:58 PM
> > Subject: [meu] Adding fields to a database
> >
> > > I'm a newcomer to programming Miva databases. All my previous
> > > experience was on flat files.
> > >
> > > So my first effort to create, add, and delete records has been
> > > successful, but now I see that having some sort of serial number on
> > > each record might make it easier to do editing and/or deleting of a
> > > specific record. Is it possible to add a new field, or do I have to
> > > go through the creation process all over again? If so, what happens
> > > to the data I've already collected?
> > >
> > > Are there approved ways of doing this sort of thing?
> > >
> > > Sandy
> >

Re: MvLOCKFILE (was: Adding fields to a database )



Hi All,

I'm cross-posting this thread because I feel it's relative not only to
MivaScript developers,
but to Merchant developers and users as well.

In summary, the MvLOCKFILE tag has some issues, which could be responsible
for occasional duplicate order numbers in Merchant.
The problem existed in Merchant 4.13 (we could see the db.mv code).
Whether or not this issue has been corrected in compiled versions is unknown.

-Brian
Burns Enterprises


On Tue, 22 Feb 2005 11:06:27 -0500, Brian(Burns Enterprises)
<[email protected]> wrote:
> Interesting. I missed this from the docs:
>
> "However, if the second process does not issue an <MvLOCKFILE> before
> attempting to access the file in question, there is nothing to prevent
> this access from taking place. For this reason, we refer to the action
> of <MvLOCKFILE> as a lock request, rather than a lock: it is a request
> to other Miva Script processes that they 'respect' the current
> process's request for exclusive access. Another way of looking at this
> is: <MvLOCKFILE> does not interact with any other Miva Script tag,
> only other <MvLOCKFILE>s."
>
> So, the server/OS/filesystem is not aware of the "lock".
>
> However, this does not explain the previous problem, where MvLOCKFILE was being
> used by both scripts.
>
> Merchant was still incorrect as of 4.13
>
> <MvFUNCTION NAME="Key_Generate" PARAMETERS="type" STANDARDOUTPUTLEVEL="">
> <MvASSIGN NAME="l.key" VALUE=0>
> <MvIF EXPR="{_DB_Open_Keys()}">
> <MvLOCKFILE FILE="{g.MerchantPath $ 'keys.dbf'}">
> <MvFIND NAME="Keys" VALUE="{l.type}" EXACT>
> <MvIF EXPR="{NOT Keys.d.EOF}">
> <MvASSIGN NAME="Keys.d.maxvalue" VALUE="{Keys.d.maxvalue + 1}">
> <MvUPDATE NAME="Keys">
> <MvASSIGN NAME="l.key" VALUE="{Keys.d.maxvalue}">
> </MvIF>
> </MvLOCKFILE>
> </MvIF>
> <MvFUNCTIONRETURN VALUE="{l.key}">
> </MvFUNCTION>
>
> The only way I've found to create keys for a unique index is:
>
> <MvFUNCTION NAME="Key_Generate" PARAMETERS="name" STANDARDOUTPUTLEVEL="">
> <MvCOMMENT>1/4/2004 -Burns</MvCOMMENT>
> <MvASSIGN NAME="l.dir" VALUE="{Data_Dir()}">
> <MvLOCKFILE FILE="{l.dir $ 'keys.dbf'}">
> <MvOPEN NAME="KeysDB" DATABASE="{l.dir $ 'keys.dbf'}"
> INDEXES="{l.dir $ 'keys.mvx'}">
> <MvIF EXPR="{NOT MvOPEN_Error}">
> <MvFIND NAME="KeysDB" VALUE="{l.name}" EXACT>
> <MvIF EXPR="{NOT KeysDB.d.eof}">
> <MvASSIGN NAME="l.key" VALUE="{KeysDB.d.value + 1}">
> <MvASSIGN NAME="KeysDB.d.value" VALUE="{l.key}">
> <MvUPDATE NAME="KeysDB">
> </MvIF>
> <MvCLOSE NAME="KeysDB">
> </MvIF>
> </MvLOCKFILE>
> <MvFUNCRETURN VALUE="{l.key}">
> </MvFUNCTION>
>
>
> On Mon, 21 Feb 2005 16:40:39 -0800, Larry Hiscock
> <[email protected]> wrote:
> > Script #2 should attempt to lock the file before it deletes it. Under
> > Unix/Linux at least, a file can be deleted from underneath a process that
> > has it open.
> >
> > Larry Hiscock
> > AngelicHost
> >
> > -----Original Message-----
> > From: [email protected] [mailto:[email protected]] On Behalf
> > Of Brian(Burns Enterprises)
> > Sent: Monday, February 21, 2005 5:09 AM
> > To: MvMarkus
> > Cc: [email protected]
> > Subject: [meu] Re: MvLOCKFILE (was: Adding fields to a database )
> >
> > Thanks for the reply Markus!
> > I certainly appreaciate your 'thinking out load' on this.
> > You make some very interesting points.
> >
> > One other interesting test I performed just recently.....
> >
> > Script #1:
> > - Lock file
> > - re-create file (using MvExport or MvCreate)
> > - loop until timeout
> >
> > Scipt #2:
> > - while #1 is in loop, try to delete the file
> >
> > Well, it deleted it. Leaving nothing but the .lck file in the directory,
> > which disappeared once the script timed out.
> >
> > Ran into this while looking at a rebuilding a database.
> > I assumed that if used MvLockfile that Script #2 would timeout waiting for
> > the lock to release.
> >
> > There definately does appear to be some error in design.
> > In fact, I believe these issues are why every Merchant store will eventually
> > will suffer from the "duplicate order_id" error we all deal with from time
> > to time.
> >
> > On Mon, 21 Feb 2005 11:57:12 +0100, MvMarkus <[email protected]> wrote:
> > > Dear Bill, Brian,
> > >
> > > (I post this back to the mailing list, maybe other users can look into
> > > it as
> > > well)
> > >
> > > I just tested the scripts last night (on a Windows 2003 server / Miva
> > > Mia
> > > (uncompiled) and EmpresaVM and uncompiled on FreeBSD (sorry I was too
> > > lazy to also test it there compiled ). The Windows server has 1
> > > processor, the FreeBSD 2.
> > >
> > > As expected I was able to reproduce the errors on both machines and
> > > versions. Interesting is that Mia and Empresa occasionally showed
> > > different results in some tests.
> > >
> > > Some thinking aloud:
> > >
> > > My uneducated guess is that this is a threading/timing and/or -as you
> > > suspected- a memory issue. One process (thread) should lock the memory
> > > access and update the table and the index. But instead MvLOCKFILE
> > > appears to only work on the filesystem or it fails to "inform" other
> > > threads/processes that the data in memory has been updated.
> > >
> > > Empresa is not a "true" server program, but merely a sophisticated
> > > cgi-application, meaning that it does not really control the dataflow
> > > of parallel processes and memory access. Instead the operating system
> > > controls it.
> > >
> > > Empresa reloads different instances of tables into different areas of
> > > the memory, which probably don't even know of each other because they
> > > are controlled by different threads. So when thread 1 locks its
> > > memoryblock (and the original file), this does -at least temporarily-
> > > not affect thread 2, since the processors are so fast that they switch
> > > to thread 2 and while 1 has not had the time to flush the data to the
> > disk.
> > >
> > > Should MvLOCKFILE indeed update the RAM, then there is also the
> > > possibility that the table sits fully in the processor cache (after
> > > all, the tables are small enough to fit into it), and MvLOCKFILE
> > > doesn't have the time to update the RAM and disk before the second
> > > thread kicks in. For this it would be interesting to see if the same
> > > phenomenon happens also with larger tables (a few megabytes or so).
> > >
> > > There is also the possibility that actually the disk cache is to blame.
> > > Earlier hard disk hardly had any, today all drives have at least 2MB -
> > > so more than enough to reload a cached table while the original (now
> > > updated) data is still written to the disk.
> > >
> > > It is also interesting to observe that when you put MvLOCKFILE ->
> > > before <- the wait() in test 1 and 2 (I didn't try the other ones),
> > > the test performs correctly, without errors. This could support the
> > theories about the timing.
> > >
> > > If any of this is indeed the case, this could be that the original
> > > design might be a few years old, where neither processors, RAM nor
> > > disk were even remotely as fast as they are today. If this hasn't been
> > > updated ever since, then timing issues like these are not entirely
> > surprising.
> > >
> > > I'm aftraid that this is not a bug, but a design error.
> > >
> > > This has obviously some very important implications, and I think I
> > > have seen this issue in real life applications, not only with
> > > adding/updating but also with packing. I don't know if they are related.
> > >
> > > I've always struggled with occasional index corruptions on some
> > > tables, and that very reason (without having had such a clear test at
> > > hand), I always add additional safeguards if possible, and also close
> > > usually tables after packing. But looking at it so clearly, these
> > > tests are very illuminating when you design applications that are
> > > supposed to run under really heavy load.
> > >
> > > Great work, Brian!
> > >
> > > Markus
> > >
> > > -----Original Message-----
> > > From: Bill Guindon [mailto:[email protected]]
> > > Sent: Sonntag, 20. Februar 2005 21:18
> > > To: MvMarkus
> > > Subject: Fwd: Adding fields to a database
> > >
> > > subject modified to dodge your filters (I hope)
> > >
> > > Just want to make sure you saw this, and I'd be interested on your
> > > take on it. Brian and I have been looking at this for quite a while.
> > >
> > > SUMMARY: MvLOCKFILE does not work unless you lock the file before you
> > > open it, and unlock it after you close it.
> > >
> > > It seems that the data is either not written to the cache, or the
> > > file, until the close happens, so any reads done in between are not
> > reliable.
> > >
> > > ---------- Forwarded message ----------
> > > From: Brian Burns <[email protected]>
> > > Date: Sat, 19 Feb 2005 21:30:36 -0500
> > > Subject: [meu] Adding fields to a database
> > > To: [email protected]
> > >
> > > I actually spent quite some time researching a bug in MvLOCKFILE.
> > > I've posted this before, but never received a response.
> > > <A HREF ="http://www.mvdevelopment.com/mvlists2/10/88999.html">http://www.mvdevelopment.com/mvlists2/10/88999.html</A>
> > >
> > > I've posted the test script I used here:
> > > <A HREF ="http://www.becoded.com/lockfile_test.txt">http://www.becoded.com/lockfile_test.txt</A>
> > >
> > > To my knowledge, the problem I encountered still exists.
> > >
> > > -Brian
> > > Burns Enterprises
> > >
> > > From: MvMarkus <[email protected]>
> > > Date: Mon, 14 Feb 2005 20:37:44 +0100
> > > Subject: RE: [meu] Adding fields to a database
> > > To: Patrick Locke <[email protected]>, Sandy Rozhon
> > > <[email protected]>, Miva Userlist <[email protected]>
> > >
> > > Unless you have an index on the serial column, a relatively secure and
> > > easy way to increment a record is to rely on the physical order of the
> > > records in the db. For this, you don't use an index.
> > >
> > > MvOpen DB -> without <- any indices
> > > MvLockfile
> > > MvGO db row=bottom
> > > MvASSIGN =l.lastID VALUE=d.serial_column
> > > MvSetIndex all indices
> > > MvASSIGN =d.serial_column Value=l.lastID+1
> > > MvASSIGN = all other columns....
> > > MvADD
> > > /MvLockfile
> > >
> > > One should not rely on the built-in locking mechanism of
> > > MvADD/MvUPDATE and instead lock the table manually with MvLOCKFILE,
> > > because the function (especially opening the indices) might be quite I/O
> > expensive (slow).
> > >
> > > Obviously, this technique only works as long as the table is not
> > > physically reorganized. In that case, it's better to use an index on
> > > d.serial_column and/or do a lookup to check if the value already
> > > exists before adding a new one.
> > >
> > > Markus
> > >
> > > -----Original Message-----
> > > From: [email protected] [mailto:[email protected]] On
> > > Behalf Of Patrick Locke
> > > Sent: Montag, 14. Februar 2005 20:31
> > > To: Sandy Rozhon; Miva Userlist
> > > Subject: Re: [meu] Adding fields to a database
> > >
> > > I am not familiar with the database you are building but if it is not
> > > updated by many people you can use alias.d.recno for this. I have
> > > some dbs that I am the only one that updates the information and i use
> > > this method all the time to update information. But with multiple
> > > people being able to update, it sometimes can cause problems.
> > >
> > > Patrick
> > > ----- Original Message -----
> > > From: "Sandy Rozhon" <[email protected]>
> > > To: "Miva Userlist" <[email protected]>
> > > Sent: Saturday, February 12, 2005 1:58 PM
> > > Subject: [meu] Adding fields to a database
> > >
> > > > I'm a newcomer to programming Miva databases. All my previous
> > > > experience was on flat files.
> > > >
> > > > So my first effort to create, add, and delete records has been
> > > > successful, but now I see that having some sort of serial number on
> > > > each record might make it easier to do editing and/or deleting of a
> > > > specific record. Is it possible to add a new field, or do I have to
> > > > go through the creation process all over again? If so, what happens
> > > > to the data I've already collected?
> > > >
> > > > Are there approved ways of doing this sort of thing?
> > > >
> > > > Sandy
> > >

Quanta and DTD



These two subjects go hand in hand so I'll post them
together.

In some other coding projects I'm doing (non-MivaScript)
I've been using Quanta Plus and really enjoy it. Lots of
neat features like function folding, auto-complete, etc.

I've searched high and low trying to find a DTEP file built
for MivaScript. Basically the file that contains all the
rule of how the language works. So if anyone else is using
Quanta for development and has a DTEP file I'd appreciate
if you could send me a copy.

Barring that, Quanta does have in import feature. It will
import a DTD file and convert it into a DTEP file.
Searching Miva's website the only DTD I can find for the
language is for version 3.15. Positively ancient. Trying to
import that gives me an error. So, if I can't get the DTEP
right off the bat, does anyone know where to get a current
copy of the DTD for MivaScript 3.97 and 4.xx?

/ScottMc

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Re: MvLOCKFILE (was: Adding fields to a database )



Greetings All,

It was my understanding that in order for MvLOCKFILE to work like one =
would
think it should, you would have to use it each and every time you opened =
a
database, whether it needed to be locked or not, in order for the ones =
that
really needed it locked, to work. I hardly ever use it, seems to be more
trouble than it's worth. My big question is since this is the case, why =
not
make Miva lock by default when opening a database and unlock when =
closed?
Maybe I'm showing my ignorance in this regard, but it would seem better =
than
having to code it (MvLOCKFILE) manually every time.

Bill M.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On =
Behalf
Of Brian(Burns Enterprises)
Sent: Tuesday, February 22, 2005 11:06 AM
To: Larry Hiscock
Cc: MvMarkus; [email protected]
Subject: Re: [meu] Re: MvLOCKFILE (was: Adding fields to a database )


Interesting. I missed this from the docs:

"However, if the second process does not issue an <MvLOCKFILE> before
attempting to access the file in question, there is nothing to prevent =
this
access from taking place. For this reason, we refer to the action of
<MvLOCKFILE> as a lock request, rather than a lock: it is a request to =
other
Miva Script processes that they 'respect' the current process's request =
for
exclusive access. Another way of looking at this
is: <MvLOCKFILE> does not interact with any other Miva Script tag, only
other <MvLOCKFILE>s."

So, the server/OS/filesystem is not aware of the "lock".

However, this does not explain the previous problem, where MvLOCKFILE =
was
being used by both scripts.

Merchant was still incorrect as of 4.13

<MvFUNCTION NAME=3D"Key_Generate" PARAMETERS=3D"type" =
STANDARDOUTPUTLEVEL=3D"">
<MvASSIGN NAME=3D"l.key" VALUE=3D0>
<MvIF EXPR=3D"{_DB_Open_Keys()}">
<MvLOCKFILE FILE=3D"{g.MerchantPath $ 'keys.dbf'}">
<MvFIND NAME=3D"Keys" VALUE=3D"{l.type}" EXACT>
<MvIF EXPR=3D"{NOT Keys.d.EOF}">
<MvASSIGN NAME=3D"Keys.d.maxvalue" =
VALUE=3D"{Keys.d.maxvalue +
1}">
<MvUPDATE NAME=3D"Keys">
<MvASSIGN NAME=3D"l.key" VALUE=3D"{Keys.d.maxvalue}">
</MvIF>
</MvLOCKFILE>
</MvIF>
<MvFUNCTIONRETURN VALUE=3D"{l.key}">
</MvFUNCTION>


The only way I've found to create keys for a unique index is:

<MvFUNCTION NAME=3D"Key_Generate" PARAMETERS=3D"name" =
STANDARDOUTPUTLEVEL=3D"">
<MvCOMMENT>1/4/2004 -Burns</MvCOMMENT>
<MvASSIGN NAME=3D"l.dir" VALUE=3D"{Data_Dir()}">
<MvLOCKFILE FILE=3D"{l.dir $ 'keys.dbf'}">
<MvOPEN NAME=3D"KeysDB" DATABASE=3D"{l.dir $ 'keys.dbf'}"
INDEXES=3D"{l.dir $ 'keys.mvx'}">
<MvIF EXPR=3D"{NOT MvOPEN_Error}">
<MvFIND NAME=3D"KeysDB" VALUE=3D"{l.name}" EXACT>
<MvIF EXPR=3D"{NOT KeysDB.d.eof}">
<MvASSIGN NAME=3D"l.key" VALUE=3D"{KeysDB.d.value + 1}">
<MvASSIGN NAME=3D"KeysDB.d.value" VALUE=3D"{l.key}">
<MvUPDATE NAME=3D"KeysDB">
</MvIF>
<MvCLOSE NAME=3D"KeysDB">
</MvIF>
</MvLOCKFILE>
<MvFUNCRETURN VALUE=3D"{l.key}">
</MvFUNCTION>



On Mon, 21 Feb 2005 16:40:39 -0800, Larry Hiscock =
<[email protected]>
wrote:
> Script #2 should attempt to lock the file before it deletes it. Under =

> Unix/Linux at least, a file can be deleted from underneath a process=20
> that has it open.
>=20
> Larry Hiscock
> AngelicHost
>=20
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On=20
> Behalf Of Brian(Burns Enterprises)
> Sent: Monday, February 21, 2005 5:09 AM
> To: MvMarkus
> Cc: [email protected]
> Subject: [meu] Re: MvLOCKFILE (was: Adding fields to a database )
>=20
> Thanks for the reply Markus!
> I certainly appreaciate your 'thinking out load' on this.
> You make some very interesting points.
>=20
> One other interesting test I performed just recently.....
>=20
> Script #1:
> - Lock file
> - re-create file (using MvExport or MvCreate)
> - loop until timeout
>=20
> Scipt #2:
> - while #1 is in loop, try to delete the file
>=20
> Well, it deleted it. Leaving nothing but the .lck file in the=20
> directory, which disappeared once the script timed out.
>=20
> Ran into this while looking at a rebuilding a database.
> I assumed that if used MvLockfile that Script #2 would timeout waiting =

> for the lock to release.
>=20
> There definately does appear to be some error in design.
> In fact, I believe these issues are why every Merchant store will=20
> eventually will suffer from the "duplicate order_id" error we all deal =

> with from time to time.
>=20
> On Mon, 21 Feb 2005 11:57:12 +0100, MvMarkus <[email protected]>=20
> wrote:
> > Dear Bill, Brian,
> >
> > (I post this back to the mailing list, maybe other users can look=20
> > into it as
> > well)
> >
> > I just tested the scripts last night (on a Windows 2003 server /=20
> > Miva Mia
> > (uncompiled) and EmpresaVM and uncompiled on FreeBSD (sorry I was=20
> > too lazy to also test it there compiled ). The Windows server has 1=20
> > processor, the FreeBSD 2.
> >
> > As expected I was able to reproduce the errors on both machines and=20
> > versions. Interesting is that Mia and Empresa occasionally showed=20
> > different results in some tests.
> >
> > Some thinking aloud:
> >
> > My uneducated guess is that this is a threading/timing and/or -as=20
> > you
> > suspected- a memory issue. One process (thread) should lock the =
memory
> > access and update the table and the index. But instead MvLOCKFILE
> > appears to only work on the filesystem or it fails to "inform" other
> > threads/processes that the data in memory has been updated.
> >
> > Empresa is not a "true" server program, but merely a sophisticated=20
> > cgi-application, meaning that it does not really control the=20
> > dataflow of parallel processes and memory access. Instead the=20
> > operating system controls it.
> >
> > Empresa reloads different instances of tables into different areas=20
> > of the memory, which probably don't even know of each other because=20
> > they are controlled by different threads. So when thread 1 locks its =

> > memoryblock (and the original file), this does -at least=20
> > temporarily- not affect thread 2, since the processors are so fast=20
> > that they switch to thread 2 and while 1 has not had the time to=20
> > flush the data to the
> disk.
> >
> > Should MvLOCKFILE indeed update the RAM, then there is also the=20
> > possibility that the table sits fully in the processor cache (after=20
> > all, the tables are small enough to fit into it), and MvLOCKFILE=20
> > doesn't have the time to update the RAM and disk before the second=20
> > thread kicks in. For this it would be interesting to see if the same =

> > phenomenon happens also with larger tables (a few megabytes or so).
> >
> > There is also the possibility that actually the disk cache is to=20
> > blame. Earlier hard disk hardly had any, today all drives have at=20
> > least 2MB - so more than enough to reload a cached table while the=20
> > original (now
> > updated) data is still written to the disk.
> >
> > It is also interesting to observe that when you put MvLOCKFILE ->=20
> > before <- the wait() in test 1 and 2 (I didn't try the other ones),=20
> > the test performs correctly, without errors. This could support the
> theories about the timing.
> >
> > If any of this is indeed the case, this could be that the original=20
> > design might be a few years old, where neither processors, RAM nor=20
> > disk were even remotely as fast as they are today. If this hasn't=20
> > been updated ever since, then timing issues like these are not=20
> > entirely
> surprising.
> >
> > I'm aftraid that this is not a bug, but a design error.
> >
> > This has obviously some very important implications, and I think I=20
> > have seen this issue in real life applications, not only with=20
> > adding/updating but also with packing. I don't know if they are =20
> > related.
> >
> > I've always struggled with occasional index corruptions on some=20
> > tables, and that very reason (without having had such a clear test=20
> > at hand), I always add additional safeguards if possible, and also=20
> > close usually tables after packing. But looking at it so clearly,=20
> > these tests are very illuminating when you design applications that=20
> > are supposed to run under really heavy load.
> >
> > Great work, Brian!
> >
> > Markus
> >
> > -----Original Message-----
> > From: Bill Guindon [mailto:[email protected]]
> > Sent: Sonntag, 20. Februar 2005 21:18
> > To: MvMarkus
> > Subject: Fwd: Adding fields to a database
> >
> > subject modified to dodge your filters (I hope)
> >
> > Just want to make sure you saw this, and I'd be interested on your=20
> > take on it. Brian and I have been looking at this for quite a=20
> > while.
> >
> > SUMMARY: MvLOCKFILE does not work unless you lock the file before=20
> > you open it, and unlock it after you close it.
> >
> > It seems that the data is either not written to the cache, or the=20
> > file, until the close happens, so any reads done in between are not
> reliable.
> >
> > ---------- Forwarded message ----------
> > From: Brian Burns <[email protected]>
> > Date: Sat, 19 Feb 2005 21:30:36 -0500
> > Subject: [meu] Adding fields to a database
> > To: [email protected]
> >
> > I actually spent quite some time researching a bug in MvLOCKFILE.=20
> > I've posted this before, but never received a response.=20
> > <A HREF ="http://www.mvdevelopment.com/mvlists2/10/88999.html">http://www.mvdevelopment.com/mvlists2/10/88999.html</A>
> >
> > I've posted the test script I used here:=20
> > <A HREF ="http://www.becoded.com/lockfile_test.txt">http://www.becoded.com/lockfile_test.txt</A>
> >
> > To my knowledge, the problem I encountered still exists.
> >
> > -Brian
> > Burns Enterprises
> >
> > From: MvMarkus <[email protected]>
> > Date: Mon, 14 Feb 2005 20:37:44 +0100
> > Subject: RE: [meu] Adding fields to a database
> > To: Patrick Locke <[email protected]>, Sandy Rozhon=20
> > <[email protected]>, Miva Userlist <[email protected]>
> >
> > Unless you have an index on the serial column, a relatively secure=20
> > and easy way to increment a record is to rely on the physical order=20
> > of the records in the db. For this, you don't use an index.
> >
> > MvOpen DB -> without <- any indices
> > MvLockfile
> > MvGO db row=3Dbottom
> > MvASSIGN =3Dl.lastID VALUE=3Dd.serial_column
> > MvSetIndex all indices
> > MvASSIGN =3Dd.serial_column Value=3Dl.lastID+1
> > MvASSIGN =3D all other columns....
> > MvADD
> > /MvLockfile
> >
> > One should not rely on the built-in locking mechanism of=20
> > MvADD/MvUPDATE and instead lock the table manually with MvLOCKFILE,=20
> > because the function (especially opening the indices) might be quite =

> > I/O
> expensive (slow).
> >
> > Obviously, this technique only works as long as the table is not=20
> > physically reorganized. In that case, it's better to use an index on =

> > d.serial_column and/or do a lookup to check if the value already=20
> > exists before adding a new one.
> >
> > Markus
> >
> > -----Original Message-----
> > From: [email protected] [mailto:[email protected]]=20
> > On Behalf Of Patrick Locke
> > Sent: Montag, 14. Februar 2005 20:31
> > To: Sandy Rozhon; Miva Userlist
> > Subject: Re: [meu] Adding fields to a database
> >
> > I am not familiar with the database you are building but if it is=20
> > not updated by many people you can use alias.d.recno for this. I=20
> > have some dbs that I am the only one that updates the information=20
> > and i use this method all the time to update information. But with=20
> > multiple people being able to update, it sometimes can cause=20
> > problems.
> >
> > Patrick
> > ----- Original Message -----
> > From: "Sandy Rozhon" <[email protected]>
> > To: "Miva Userlist" <[email protected]>
> > Sent: Saturday, February 12, 2005 1:58 PM
> > Subject: [meu] Adding fields to a database
> >
> > > I'm a newcomer to programming Miva databases. All my previous=20
> > > experience was on flat files.
> > >
> > > So my first effort to create, add, and delete records has been=20
> > > successful, but now I see that having some sort of serial number=20
> > > on each record might make it easier to do editing and/or deleting=20
> > > of a specific record. Is it possible to add a new field, or do I=20
> > > have to go through the creation process all over again? If so,=20
> > > what happens to the data I've already collected?
> > >
> > > Are there approved ways of doing this sort of thing?
> > >
> > > Sandy
> >

Re: MvLOCKFILE (was: Adding fields to a database )



Hi Bill,

I certainly wouldn't like that, after all it creates these pesky little
.lck-files, so every simple read access of a table would also imply a write
operation.

MvADD and MvUPDATE are supposed to lock the table automatically. The
Mivascript reference manual v.3.7/9 states:

"As indicated in the database sections, when you are adding (<MvADD>) or
updating (<MvUPDATE>) a database record, Miva automatically prevents the
same record from being updated by more than one process at the same time. No
other Miva Script process can access a record while it is locked. "

... And in the database section:

"Database and index files are locked at the record level automatically when
records are added or updated to prevent data corruption (that is, two
programs cannot update the same record at the same time). Entire files can
be locked using <MvLOCKFILE>."

Something else that is interesting to consider:
"Multiple lock requests with <MvLOCKFILE> on the same file are queued. There
is no limit on the number of files on which lock requests can be made.
filename can be any file--a database, memo, index, or flat (.dat,.txt, etc.)
file. <MvLOCKFILE> tags can be nested."

They explicitly recommend to lock BOTH the database file AND the index:"
...you should lock both the database and index files in question... ".

But then: "However, it can not be guaranteed which update or add will occur
first." Outch.

I now wonder how the implicit record-level locks of MvUPDATE/MvADD affect or
are affected by the file level MvLOCKFILE... Puzzling stuff. Then pair all
this with possible operating and hardware-related memory/timing issues, and
we have a nice non-deterministic programming model.

Markus



P.S. I tried Brian's second test and locked the index as well, and it still
returned the error.

P.P.S. Talking about all this makes me miss our old friend Jack. He would
have loved to discuss this in greatest length, and by now he probably had
already revered-engineered the entire internal mechanics of MvLockfile.

*****







-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Bill Matlock
Sent: Dienstag, 22. Februar 2005 22:47
To: Miva Users List
Subject: RE: [meu] Re: MvLOCKFILE (was: Adding fields to a database )

Greetings All,

It was my understanding that in order for MvLOCKFILE to work like one would
think it should, you would have to use it each and every time you opened a
database, whether it needed to be locked or not, in order for the ones that
really needed it locked, to work. I hardly ever use it, seems to be more
trouble than it's worth. My big question is since this is the case, why not
make Miva lock by default when opening a database and unlock when closed?
Maybe I'm showing my ignorance in this regard, but it would seem better than
having to code it (MvLOCKFILE) manually every time.

Bill M.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Brian(Burns Enterprises)
Sent: Tuesday, February 22, 2005 11:06 AM
To: Larry Hiscock
Cc: MvMarkus; [email protected]
Subject: Re: [meu] Re: MvLOCKFILE (was: Adding fields to a database )


Interesting. I missed this from the docs:

"However, if the second process does not issue an <MvLOCKFILE> before
attempting to access the file in question, there is nothing to prevent this
access from taking place. For this reason, we refer to the action of
<MvLOCKFILE> as a lock request, rather than a lock: it is a request to other
Miva Script processes that they 'respect' the current process's request for
exclusive access. Another way of looking at this
is: <MvLOCKFILE> does not interact with any other Miva Script tag, only
other <MvLOCKFILE>s."

So, the server/OS/filesystem is not aware of the "lock".

However, this does not explain the previous problem, where MvLOCKFILE was
being used by both scripts.

Merchant was still incorrect as of 4.13

<MvFUNCTION NAME="Key_Generate" PARAMETERS="type" STANDARDOUTPUTLEVEL="">
<MvASSIGN NAME="l.key" VALUE=0>
<MvIF EXPR="{_DB_Open_Keys()}">
<MvLOCKFILE FILE="{g.MerchantPath $ 'keys.dbf'}">
<MvFIND NAME="Keys" VALUE="{l.type}" EXACT>
<MvIF EXPR="{NOT Keys.d.EOF}">
<MvASSIGN NAME="Keys.d.maxvalue" VALUE="{Keys.d.maxvalue +
1}">
<MvUPDATE NAME="Keys">
<MvASSIGN NAME="l.key" VALUE="{Keys.d.maxvalue}">
</MvIF>
</MvLOCKFILE>
</MvIF>
<MvFUNCTIONRETURN VALUE="{l.key}">
</MvFUNCTION>


The only way I've found to create keys for a unique index is:

<MvFUNCTION NAME="Key_Generate" PARAMETERS="name" STANDARDOUTPUTLEVEL="">
<MvCOMMENT>1/4/2004 -Burns</MvCOMMENT>
<MvASSIGN NAME="l.dir" VALUE="{Data_Dir()}">
<MvLOCKFILE FILE="{l.dir $ 'keys.dbf'}">
<MvOPEN NAME="KeysDB" DATABASE="{l.dir $ 'keys.dbf'}"
INDEXES="{l.dir $ 'keys.mvx'}">
<MvIF EXPR="{NOT MvOPEN_Error}">
<MvFIND NAME="KeysDB" VALUE="{l.name}" EXACT>
<MvIF EXPR="{NOT KeysDB.d.eof}">
<MvASSIGN NAME="l.key" VALUE="{KeysDB.d.value + 1}">
<MvASSIGN NAME="KeysDB.d.value" VALUE="{l.key}">
<MvUPDATE NAME="KeysDB">
</MvIF>
<MvCLOSE NAME="KeysDB">
</MvIF>
</MvLOCKFILE>
<MvFUNCRETURN VALUE="{l.key}">
</MvFUNCTION>



On Mon, 21 Feb 2005 16:40:39 -0800, Larry Hiscock <[email protected]>
wrote:
> Script #2 should attempt to lock the file before it deletes it. Under
> Unix/Linux at least, a file can be deleted from underneath a process
> that has it open.
>
> Larry Hiscock
> AngelicHost
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On
> Behalf Of Brian(Burns Enterprises)
> Sent: Monday, February 21, 2005 5:09 AM
> To: MvMarkus
> Cc: [email protected]
> Subject: [meu] Re: MvLOCKFILE (was: Adding fields to a database )
>
> Thanks for the reply Markus!
> I certainly appreaciate your 'thinking out load' on this.
> You make some very interesting points.
>
> One other interesting test I performed just recently.....
>
> Script #1:
> - Lock file
> - re-create file (using MvExport or MvCreate)
> - loop until timeout
>
> Scipt #2:
> - while #1 is in loop, try to delete the file
>
> Well, it deleted it. Leaving nothing but the .lck file in the
> directory, which disappeared once the script timed out.
>
> Ran into this while looking at a rebuilding a database.
> I assumed that if used MvLockfile that Script #2 would timeout waiting
> for the lock to release.
>
> There definately does appear to be some error in design.
> In fact, I believe these issues are why every Merchant store will
> eventually will suffer from the "duplicate order_id" error we all deal
> with from time to time.
>
> On Mon, 21 Feb 2005 11:57:12 +0100, MvMarkus <[email protected]>
> wrote:
> > Dear Bill, Brian,
> >
> > (I post this back to the mailing list, maybe other users can look
> > into it as
> > well)
> >
> > I just tested the scripts last night (on a Windows 2003 server /
> > Miva Mia
> > (uncompiled) and EmpresaVM and uncompiled on FreeBSD (sorry I was
> > too lazy to also test it there compiled ). The Windows server has 1
> > processor, the FreeBSD 2.
> >
> > As expected I was able to reproduce the errors on both machines and
> > versions. Interesting is that Mia and Empresa occasionally showed
> > different results in some tests.
> >
> > Some thinking aloud:
> >
> > My uneducated guess is that this is a threading/timing and/or -as
> > you
> > suspected- a memory issue. One process (thread) should lock the
> > memory access and update the table and the index. But instead
> > MvLOCKFILE appears to only work on the filesystem or it fails to
> > "inform" other threads/processes that the data in memory has been
updated.
> >
> > Empresa is not a "true" server program, but merely a sophisticated
> > cgi-application, meaning that it does not really control the
> > dataflow of parallel processes and memory access. Instead the
> > operating system controls it.
> >
> > Empresa reloads different instances of tables into different areas
> > of the memory, which probably don't even know of each other because
> > they are controlled by different threads. So when thread 1 locks its
> > memoryblock (and the original file), this does -at least
> > temporarily- not affect thread 2, since the processors are so fast
> > that they switch to thread 2 and while 1 has not had the time to
> > flush the data to the
> disk.
> >
> > Should MvLOCKFILE indeed update the RAM, then there is also the
> > possibility that the table sits fully in the processor cache (after
> > all, the tables are small enough to fit into it), and MvLOCKFILE
> > doesn't have the time to update the RAM and disk before the second
> > thread kicks in. For this it would be interesting to see if the same
> > phenomenon happens also with larger tables (a few megabytes or so).
> >
> > There is also the possibility that actually the disk cache is to
> > blame. Earlier hard disk hardly had any, today all drives have at
> > least 2MB - so more than enough to reload a cached table while the
> > original (now
> > updated) data is still written to the disk.
> >
> > It is also interesting to observe that when you put MvLOCKFILE ->
> > before <- the wait() in test 1 and 2 (I didn't try the other ones),
> > the test performs correctly, without errors. This could support the
> theories about the timing.
> >
> > If any of this is indeed the case, this could be that the original
> > design might be a few years old, where neither processors, RAM nor
> > disk were even remotely as fast as they are today. If this hasn't
> > been updated ever since, then timing issues like these are not
> > entirely
> surprising.
> >
> > I'm aftraid that this is not a bug, but a design error.
> >
> > This has obviously some very important implications, and I think I
> > have seen this issue in real life applications, not only with
> > adding/updating but also with packing. I don't know if they are
> > related.
> >
> > I've always struggled with occasional index corruptions on some
> > tables, and that very reason (without having had such a clear test
> > at hand), I always add additional safeguards if possible, and also
> > close usually tables after packing. But looking at it so clearly,
> > these tests are very illuminating when you design applications that
> > are supposed to run under really heavy load.
> >
> > Great work, Brian!
> >
> > Markus
> >
> > -----Original Message-----
> > From: Bill Guindon [mailto:[email protected]]
> > Sent: Sonntag, 20. Februar 2005 21:18
> > To: MvMarkus
> > Subject: Fwd: Adding fields to a database
> >
> > subject modified to dodge your filters (I hope)
> >
> > Just want to make sure you saw this, and I'd be interested on your
> > take on it. Brian and I have been looking at this for quite a
> > while.
> >
> > SUMMARY: MvLOCKFILE does not work unless you lock the file before
> > you open it, and unlock it after you close it.
> >
> > It seems that the data is either not written to the cache, or the
> > file, until the close happens, so any reads done in between are not
> reliable.
> >
> > ---------- Forwarded message ----------
> > From: Brian Burns <[email protected]>
> > Date: Sat, 19 Feb 2005 21:30:36 -0500
> > Subject: [meu] Adding fields to a database
> > To: [email protected]
> >
> > I actually spent quite some time researching a bug in MvLOCKFILE.
> > I've posted this before, but never received a response.
> > <A HREF ="http://www.mvdevelopment.com/mvlists2/10/88999.html">http://www.mvdevelopment.com/mvlists2/10/88999.html</A>
> >
> > I've posted the test script I used here:
> > <A HREF ="http://www.becoded.com/lockfile_test.txt">http://www.becoded.com/lockfile_test.txt</A>
> >
> > To my knowledge, the problem I encountered still exists.
> >
> > -Brian
> > Burns Enterprises
> >
> > From: MvMarkus <[email protected]>
> > Date: Mon, 14 Feb 2005 20:37:44 +0100
> > Subject: RE: [meu] Adding fields to a database
> > To: Patrick Locke <[email protected]>, Sandy Rozhon
> > <[email protected]>, Miva Userlist <[email protected]>
> >
> > Unless you have an index on the serial column, a relatively secure
> > and easy way to increment a record is to rely on the physical order
> > of the records in the db. For this, you don't use an index.
> >
> > MvOpen DB -> without <- any indices
> > MvLockfile
> > MvGO db row=bottom
> > MvASSIGN =l.lastID VALUE=d.serial_column
> > MvSetIndex all indices
> > MvASSIGN =d.serial_column Value=l.lastID+1
> > MvASSIGN = all other columns....
> > MvADD
> > /MvLockfile
> >
> > One should not rely on the built-in locking mechanism of
> > MvADD/MvUPDATE and instead lock the table manually with MvLOCKFILE,
> > because the function (especially opening the indices) might be quite
> > I/O
> expensive (slow).
> >
> > Obviously, this technique only works as long as the table is not
> > physically reorganized. In that case, it's better to use an index on
> > d.serial_column and/or do a lookup to check if the value already
> > exists before adding a new one.
> >
> > Markus
> >
> > -----Original Message-----
> > From: [email protected] [mailto:[email protected]]
> > On Behalf Of Patrick Locke
> > Sent: Montag, 14. Februar 2005 20:31
> > To: Sandy Rozhon; Miva Userlist
> > Subject: Re: [meu] Adding fields to a database
> >
> > I am not familiar with the database you are building but if it is
> > not updated by many people you can use alias.d.recno for this. I
> > have some dbs that I am the only one that updates the information
> > and i use this method all the time to update information. But with
> > multiple people being able to update, it sometimes can cause
> > problems.
> >
> > Patrick
> > ----- Original Message -----
> > From: "Sandy Rozhon" <[email protected]>
> > To: "Miva Userlist" <[email protected]>
> > Sent: Saturday, February 12, 2005 1:58 PM
> > Subject: [meu] Adding fields to a database
> >
> > > I'm a newcomer to programming Miva databases. All my previous
> > > experience was on flat files.
> > >
> > > So my first effort to create, add, and delete records has been
> > > successful, but now I see that having some sort of serial number
> > > on each record might make it easier to do editing and/or deleting
> > > of a specific record. Is it possible to add a new field, or do I
> > > have to go through the creation process all over again? If so,
> > > what happens to the data I've already collected?
> > >
> > > Are there approved ways of doing this sort of thing?
> > >
> > > Sandy
> >

Re: MvLOCKFILE (was: Adding fields to a database )



On Wed, 23 Feb 2005 01:34:43 +0100, MvMarkus <[email protected]> wrote:
> Hi Bill,
>
> I certainly wouldn't like that, after all it creates these pesky little
> .lck-files, so every simple read access of a table would also imply a write
> operation.

Not to mention being highly inefficient. Imagine a blog driven by
miva. The editor "writes" to the database once a day, thousands of
users "read" from it during the day. Locking for every read would add
a lot of cpu ticks, and slow the site down (due to the file constantly
being locked).

> MvADD and MvUPDATE are supposed to lock the table automatically. The
> Mivascript reference manual v.3.7/9 states:
>
> "As indicated in the database sections, when you are adding (<MvADD>) or
> updating (<MvUPDATE>) a database record, Miva automatically prevents the
> same record from being updated by more than one process at the same time. No
> other Miva Script process can access a record while it is locked. "
>
> ... And in the database section:
>
> "Database and index files are locked at the record level automatically when
> records are added or updated to prevent data corruption (that is, two
> programs cannot update the same record at the same time). Entire files can
> be locked using <MvLOCKFILE>."

Well, that's not the file, it's just a single record. Perhaps this
would best be described as "minimal locking"? It sounds like this is
just to prevent "physical" corruption, not "logical" corruption (ie:
it's there to maintain file integrity, not accuracy).

> Something else that is interesting to consider:
> "Multiple lock requests with <MvLOCKFILE> on the same file are queued. There
> is no limit on the number of files on which lock requests can be made.
> filename can be any file--a database, memo, index, or flat (.dat,.txt, etc.)
> file. <MvLOCKFILE> tags can be nested."
>
> They explicitly recommend to lock BOTH the database file AND the index:"
> ...you should lock both the database and index files in question... ".
>
> But then: "However, it can not be guaranteed which update or add will occur
> first." Outch.

That is interesting. I don't think I have ever locked an index file.
We did discuss it though, but came to the conclusion that there was
probably no need since the database acts as a "gateway" to the index.
If it's locked, the index "shouldn't" change. Or so I thought, until
I read that.

It's also fairly well known (as Brian pointed out) that this was not a
common practice in earlier versions of Merchant. Since many of us
learned by reading the source, we picked up the habits that were
there.

> I now wonder how the implicit record-level locks of MvUPDATE/MvADD affect or
> are affected by the file level MvLOCKFILE... Puzzling stuff. Then pair all
> this with possible operating and hardware-related memory/timing issues, and
> we have a nice non-deterministic programming model.



> Markus
>
> P.S. I tried Brian's second test and locked the index as well, and it still
> returned the error.

I think Larry's right about that one, although I must admit, it's not
intuitive. Personally, I think the language should be your tool (not
the other way around), seems silly that a "delete" command (among
others) doesn't check for locks, but that's just my opinion.

> P.P.S. Talking about all this makes me miss our old friend Jack. He would
> have loved to discuss this in greatest length,

Well, we'll have to see how much we can fill in for him. Reverse
engineering of MvLockfile

> and by now he probably had
> already revered-engineered the entire internal mechanics of MvLockfile.

Hmm, wonder if there's any connection there :)

--
Bill Guindon (aka aGorilla)

Re: MvLOCKFILE (was: Adding fields to a database )



(from below)
> > Something else that is interesting to consider:
> > "Multiple lock requests with <MvLOCKFILE> on the same file are queued. There
> > is no limit on the number of files on which lock requests can be made.
> > filename can be any file--a database, memo, index, or flat (.dat,.txt, etc.)
> > file. <MvLOCKFILE> tags can be nested."

If you nest locktags like this:
- lock existing file
- mvexport over it
- try to lock the file again

You will get the error:
MvLOCKFILE: Error creating lockfile 'test/locktest.txt': Possible
deadlock detected

So, you can nest...but not on the same file.

-Brian

On Tue, 22 Feb 2005 20:29:35 -0500, Bill Guindon <[email protected]> wrote:
> On Wed, 23 Feb 2005 01:34:43 +0100, MvMarkus <[email protected]> wrote:
> > Hi Bill,
> >
> > I certainly wouldn't like that, after all it creates these pesky little
> > .lck-files, so every simple read access of a table would also imply a write
> > operation.
>
> Not to mention being highly inefficient. Imagine a blog driven by
> miva. The editor "writes" to the database once a day, thousands of
> users "read" from it during the day. Locking for every read would add
> a lot of cpu ticks, and slow the site down (due to the file constantly
> being locked).
>
> > MvADD and MvUPDATE are supposed to lock the table automatically. The
> > Mivascript reference manual v.3.7/9 states:
> >
> > "As indicated in the database sections, when you are adding (<MvADD>) or
> > updating (<MvUPDATE>) a database record, Miva automatically prevents the
> > same record from being updated by more than one process at the same time. No
> > other Miva Script process can access a record while it is locked. "
> >
> > ... And in the database section:
> >
> > "Database and index files are locked at the record level automatically when
> > records are added or updated to prevent data corruption (that is, two
> > programs cannot update the same record at the same time). Entire files can
> > be locked using <MvLOCKFILE>."
>
> Well, that's not the file, it's just a single record. Perhaps this
> would best be described as "minimal locking"? It sounds like this is
> just to prevent "physical" corruption, not "logical" corruption (ie:
> it's there to maintain file integrity, not accuracy).
>
> > Something else that is interesting to consider:
> > "Multiple lock requests with <MvLOCKFILE> on the same file are queued. There
> > is no limit on the number of files on which lock requests can be made.
> > filename can be any file--a database, memo, index, or flat (.dat,.txt, etc.)
> > file. <MvLOCKFILE> tags can be nested."
> >
> > They explicitly recommend to lock BOTH the database file AND the index:"
> > ...you should lock both the database and index files in question... ".
> >
> > But then: "However, it can not be guaranteed which update or add will occur
> > first." Outch.
>
> That is interesting. I don't think I have ever locked an index file.
> We did discuss it though, but came to the conclusion that there was
> probably no need since the database acts as a "gateway" to the index.
> If it's locked, the index "shouldn't" change. Or so I thought, until
> I read that.
>
> It's also fairly well known (as Brian pointed out) that this was not a
> common practice in earlier versions of Merchant. Since many of us
> learned by reading the source, we picked up the habits that were
> there.
>
> > I now wonder how the implicit record-level locks of MvUPDATE/MvADD affect or
> > are affected by the file level MvLOCKFILE... Puzzling stuff. Then pair all
> > this with possible operating and hardware-related memory/timing issues, and
> > we have a nice non-deterministic programming model.
>
> > Markus
> >
> > P.S. I tried Brian's second test and locked the index as well, and it still
> > returned the error.
>
> I think Larry's right about that one, although I must admit, it's not
> intuitive. Personally, I think the language should be your tool (not
> the other way around), seems silly that a "delete" command (among
> others) doesn't check for locks, but that's just my opinion.
>
> > P.P.S. Talking about all this makes me miss our old friend Jack. He would
> > have loved to discuss this in greatest length,
>
> Well, we'll have to see how much we can fill in for him. Reverse
> engineering of MvLockfile
>
> > and by now he probably had
> > already revered-engineered the entire internal mechanics of MvLockfile.
>
> Hmm, wonder if there's any connection there :)
>
> --
> Bill Guindon (aka aGorilla)
>

Re: MvLOCKFILE (was: Adding fields to a database )



<MvLOCK>
CEO =C7AKAFETE Inc, Fred White
10 High Street
ROSEAU
Dominica. W.I

URL: <A HREF ="http://www.cakafete.com/">http://www.cakafete.com/</A>
Tel: (767) 448 1712
Webmaster: 767 235 2795
</MvLOCK>


On Tue, 22 Feb 2005 20:49:02 -0500, Brian(Burns Enterprises)
<[email protected]> wrote:
> (from below)
> > > Something else that is interesting to consider:
> > > "Multiple lock requests with <MvLOCKFILE> on the same file are queued=
. There
> > > is no limit on the number of files on which lock requests can be made=
.
> > > filename can be any file--a database, memo, index, or flat (.dat,.txt=
, etc.)
> > > file. <MvLOCKFILE> tags can be nested."
>=20
> If you nest locktags like this:
> - lock existing file
> - mvexport over it
> - try to lock the file again
>=20
> You will get the error:
> MvLOCKFILE: Error creating lockfile 'test/locktest.txt': Possible
> deadlock detected
>=20
> So, you can nest...but not on the same file.
>=20
> -Brian
>=20
> On Tue, 22 Feb 2005 20:29:35 -0500, Bill Guindon <[email protected]> wro=
te:
> > On Wed, 23 Feb 2005 01:34:43 +0100, MvMarkus <[email protected]> wrot=
e:
> > > Hi Bill,
> > >
> > > I certainly wouldn't like that, after all it creates these pesky litt=
le
> > > .lck-files, so every simple read access of a table would also imply a=
write
> > > operation.
> >
> > Not to mention being highly inefficient. Imagine a blog driven by
> > miva. The editor "writes" to the database once a day, thousands of
> > users "read" from it during the day. Locking for every read would add
> > a lot of cpu ticks, and slow the site down (due to the file constantly
> > being locked).
> >
> > > MvADD and MvUPDATE are supposed to lock the table automatically. The
> > > Mivascript reference manual v.3.7/9 states:
> > >
> > > "As indicated in the database sections, when you are adding (<MvADD>)=
or
> > > updating (<MvUPDATE>) a database record, Miva automatically prevents =
the
> > > same record from being updated by more than one process at the same t=
ime. No
> > > other Miva Script process can access a record while it is locked. "
> > >
> > > ... And in the database section:
> > >
> > > "Database and index files are locked at the record level automaticall=
y when
> > > records are added or updated to prevent data corruption (that is, two
> > > programs cannot update the same record at the same time). Entire file=
s can
> > > be locked using <MvLOCKFILE>."
> >
> > Well, that's not the file, it's just a single record. Perhaps this
> > would best be described as "minimal locking"? It sounds like this is
> > just to prevent "physical" corruption, not "logical" corruption (ie:
> > it's there to maintain file integrity, not accuracy).
> >
> > > Something else that is interesting to consider:
> > > "Multiple lock requests with <MvLOCKFILE> on the same file are queued=
. There
> > > is no limit on the number of files on which lock requests can be made=
.
> > > filename can be any file--a database, memo, index, or flat (.dat,.txt=
, etc.)
> > > file. <MvLOCKFILE> tags can be nested."
> > >
> > > They explicitly recommend to lock BOTH the database file AND the inde=
x:"
> > > ...you should lock both the database and index files in question... "=
.
> > >
> > > But then: "However, it can not be guaranteed which update or add will=
occur
> > > first." Outch.
> >
> > That is interesting. I don't think I have ever locked an index file.
> > We did discuss it though, but came to the conclusion that there was
> > probably no need since the database acts as a "gateway" to the index.
> > If it's locked, the index "shouldn't" change. Or so I thought, until
> > I read that.
> >
> > It's also fairly well known (as Brian pointed out) that this was not a
> > common practice in earlier versions of Merchant. Since many of us
> > learned by reading the source, we picked up the habits that were
> > there.
> >
> > > I now wonder how the implicit record-level locks of MvUPDATE/MvADD af=
fect or
> > > are affected by the file level MvLOCKFILE... Puzzling stuff. Then pai=
r all
> > > this with possible operating and hardware-related memory/timing issue=
s, and
> > > we have a nice non-deterministic programming model.
> >
> > > Markus
> > >
> > > P.S. I tried Brian's second test and locked the index as well, and it=
still
> > > returned the error.
> >
> > I think Larry's right about that one, although I must admit, it's not
> > intuitive. Personally, I think the language should be your tool (not
> > the other way around), seems silly that a "delete" command (among
> > others) doesn't check for locks, but that's just my opinion.
> >
> > > P.P.S. Talking about all this makes me miss our old friend Jack. He w=
ould
> > > have loved to discuss this in greatest length,
> >
> > Well, we'll have to see how much we can fill in for him. Reverse
> > engineering of MvLockfile
> >
> > > and by now he probably had
> > > already revered-engineered the entire internal mechanics of MvLockfil=
e.
> >
> > Hmm, wonder if there's any connection there :)
> >
> > --
> > Bill Guindon (aka aGorilla)
> >
>=20


--=20
-Brian
Burns Enterprises

Re: MvLOCKFILE (was: Adding fields to a database )



------=_NextPart_000_0002_01C5197C.C1AE1E60
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Markus,

> I certainly wouldn't like that, after all it creates these
> pesky little .lck-files, so every simple read access of a
> table would also imply a write operation.

That's exactly my point. (kinda) In order for the simple read to comply =
with
a previous lock, then it would also need a lock. Seems kinda silly =
really.
What's the point of that? Say your packing a database that may only take =
5
seconds to complete, but if you don't want any reads to interfere, and =
since
you can't do IF PACKING THEN WAIT ELSE PROCEED, you need to have the =
lock
for all the reads in place all the time. Maybe my question should be =
"why
lock a database (or any file) if other non-locked operations are not =
going
to comply with the lock?" I guess I never really understood the need for
MvLOCKFILE. It seemed like a usefull tool at first untill I found out =
how it
really works or maybe I'm missing the point.

Bill M.


------=_NextPart_000_0002_01C5197C.C1AE1E60
Content-Type: text/plain; charset=us-ascii

Quanta and DTD



Hi Scott

I'm quanta user v3.0 on my RedHat 8.0 distro. I don't found that your DTD=
=20
said.=20


Roberto Buccino
IT Developer
______________________________
Alfa Centauro Custom System
Copahue 3480 Melipal
(02944) 44 11 88 / 15 60 3287
______________________________
San Carlos de Bariloche - Rio Negro
Patagonia - Argentina=20
<A HREF ="http://www.paralelo42.com">http://www.paralelo42.com</A>


El Martes 22 Febrero 2005 17:59, Scott McC escribi=F3:
> These two subjects go hand in hand so I'll post them
> together.
>
> In some other coding projects I'm doing (non-MivaScript)
> I've been using Quanta Plus and really enjoy it. Lots of
> neat features like function folding, auto-complete, etc.
>
> I've searched high and low trying to find a DTEP file built
> for MivaScript. Basically the file that contains all the
> rule of how the language works. So if anyone else is using
> Quanta for development and has a DTEP file I'd appreciate
> if you could send me a copy.
>
> Barring that, Quanta does have in import feature. It will
> import a DTD file and convert it into a DTEP file.
> Searching Miva's website the only DTD I can find for the
> language is for version 3.15. Positively ancient. Trying to
> import that gives me an error. So, if I can't get the DTEP
> right off the bat, does anyone know where to get a current
> copy of the DTD for MivaScript 3.97 and 4.xx?
>
> /ScottMc
>

Re: MvLOCKFILE (was: Adding fields to a database )



I think you're misunderstanding the issue.
If you lock a .dbf, then try to read/write to it from another script,
the script "will" wait until the lock is released.

"As indicated in the database sections, when you are adding (<MvADD>) or
updating (<MvUPDATE>) a database record, Miva automatically prevents the
same record from being updated by more than one process at the same time. No
other Miva Script process can access a record while it is locked. "

This locking takes place in Empressa....as Larry pointed out,
the filesytem is not aware of the lock.

Understand, concurrently modifying a dbf or other file is not the issue.
It's when the first script releases it's lock, the second script
that has been waiting to read does not see the changes that
were made by the first script.....unless you close and re-open the dbf.

As Markus pointed out, this is most likely a caching and/or timing issue.

--
-Brian
Burns Enterprises

On Wed, 23 Feb 2005 07:53:20 -0500, [email protected]
<[email protected]> wrote:
> Markus,
>
> > I certainly wouldn't like that, after all it creates these
> > pesky little .lck-files, so every simple read access of a
> > table would also imply a write operation.
>
> That's exactly my point. (kinda) In order for the simple read to comply with
> a previous lock, then it would also need a lock. Seems kinda silly really.
> What's the point of that? Say your packing a database that may only take 5
> seconds to complete, but if you don't want any reads to interfere, and since
> you can't do IF PACKING THEN WAIT ELSE PROCEED, you need to have the lock
> for all the reads in place all the time. Maybe my question should be "why
> lock a database (or any file) if other non-locked operations are not going
> to comply with the lock?" I guess I never really understood the need for
> MvLOCKFILE. It seemed like a usefull tool at first untill I found out how it
> really works or maybe I'm missing the point.
>
> Bill M.
>
>

Re: MvLOCKFILE (was: Adding fields to a database )



Brian,

> I think you're misunderstanding the issue.
> If you lock a .dbf, then try to read/write to it from another=20
> script, the script "will" wait until the lock is released.

Even if the "other" script does not use a lock? This is where I'm fuzzy =
on
the issue of MvLOCKFILE.

Quote:
"However, if the second process does not issue an <MvLOCKFILE> before
attempting to access the file in question, there is nothing to prevent =
this
access from taking place. For this reason, we refer to the action of
<MvLOCKFILE> as a lock request, rather than a lock: it is a request to =
other
Miva Script processes that they 'respect' the current process's request =
for
exclusive access. Another way of looking at this
is: <MvLOCKFILE> does not interact with any other Miva Script tag, only
other <MvLOCKFILE>s."

So, according to this, the 2nd script will need a lock in order for it =
to
wait for the 1st script's lock to be released. If the 2nd script was =
just a
simple read, it would still require a lock for it to wait for the first
script's lock to be released? And if so, my sometimes unlogical mind =
tells
me that in order to always have a lock "trully" lock a file, all =
database
operations would need to be inside a lock as well, because - =
"<MvLOCKFILE>
does not interact with any other Miva Script tag, only other =
<MvLOCKFILE>s".

Sorry if I'm confusing the issue, but this whole lockfile thing has =
always
given me fits.

Thanks,
Bill M.