MvLOCKFILE (was: Adding fields to a database )

Re: MvLOCKFILE (was: Adding fields to a database )



The second script calling does not need to use the MvLOCKFILE command.
Empresa is already performing this.

<snip>
"As indicated in the database sections, when you are adding (<MvADD>) or
updating (<MvUPDATE>) a database record, Miva automatically prevents the
same record from being updated by more than one process at the same time. No
other Miva Script process can access a record while it is locked. "

... And in the database section:

"Database and index files are locked at the record level automatically when
records are added or updated to prevent data corruption (that is, two
programs cannot update the same record at the same time). Entire files can
be locked using <MvLOCKFILE>."
</snip>

Even though the manual refers to a "record lock" and a "file lock",
I don't believe there is any difference in what Empresa is doing.

FYI: I know the caching bug affects databases
(as per the original test script <A HREF ="http://www.becoded.com/lockfile_test.txt)">http://www.becoded.com/lockfile_test.txt)</A>
I have not run these tests using read/write from a text file....so not sure if
this would be affected also.
(i.e. second script doesn't see line updated/added by first script)

-Brian
Burns Enterprises

On Wed, 23 Feb 2005 17:35:39 -0500, Bill Matlock
<[email protected]> wrote:
> Brian,
>
> > I think you're misunderstanding the issue.
> > If you lock a .dbf, then try to read/write to it from another
> > script, the script "will" wait until the lock is released.
>
> Even if the "other" script does not use a lock? This is where I'm fuzzy on
> the issue of MvLOCKFILE.
>
> Quote:
> "However, if the second process does not issue an <MvLOCKFILE> before
> attempting to access the file in question, there is nothing to prevent this
> access from taking place. For this reason, we refer to the action of
> <MvLOCKFILE> as a lock request, rather than a lock: it is a request to other
> Miva Script processes that they 'respect' the current process's request for
> exclusive access. Another way of looking at this
> is: <MvLOCKFILE> does not interact with any other Miva Script tag, only
> other <MvLOCKFILE>s."
>
> So, according to this, the 2nd script will need a lock in order for it to
> wait for the 1st script's lock to be released. If the 2nd script was just a
> simple read, it would still require a lock for it to wait for the first
> script's lock to be released? And if so, my sometimes unlogical mind tells
> me that in order to always have a lock "trully" lock a file, all database
> operations would need to be inside a lock as well, because - "<MvLOCKFILE>
> does not interact with any other Miva Script tag, only other <MvLOCKFILE>s".
>
> Sorry if I'm confusing the issue, but this whole lockfile thing has always
> given me fits.
>
> Thanks,
> Bill M.
>

Valueweb slow with OpenUI -- new host needed



Valueweb will NOT support sites with OpenUI installed (ie: if its slow
then there's nothing they can do about it) so we are bailing on Valueweb
(worthless).

Hosting recommendations? MUST SUPPORT OPENUI.

Jerry

Valueweb slow with OpenUI -- new host needed



DotComDesigners.com ... great people.

Jon Coulter
Coulter Web Services

-----Original Message-----
From: [email protected] [mailto:[email protected]]On
Behalf Of Jerome Draper
Sent: Wednesday, February 23, 2005 5:58 PM
Cc: Miva Users List
Subject: [meu] Valueweb slow with OpenUI -- new host needed


Valueweb will NOT support sites with OpenUI installed (ie: if its slow
then there's nothing they can do about it) so we are bailing on Valueweb
(worthless).

Hosting recommendations? MUST SUPPORT OPENUI.

Jerry

Valueweb slow with OpenUI -- new host needed



Jerry,

I'm hosted with Hostasaurus (as are many others "in the know") and quite
heartily recommend them. They are knowledgable about Miva Merchant, they have
speedy servers, and their prices are extremely competitive. They will move
your site for free and with no downtime. And yes, they support OpenUI. :)

HTH
Tom

> -----Original Message-----
> From: [email protected] [mailto:[email protected]]On
> Behalf Of Jerome Draper
> Sent: Wednesday, February 23, 2005 4:58 PM
> Cc: Miva Users List
> Subject: [meu] Valueweb slow with OpenUI -- new host needed
>
>
> Valueweb will NOT support sites with OpenUI installed (ie: if its slow
> then there's nothing they can do about it) so we are bailing on Valueweb
> (worthless).
>
> Hosting recommendations? MUST SUPPORT OPENUI.
>
> Jerry
>

Quanta and DTD



The steps I'm using are:
1) Go to DTD -> Load & Convert DTD
2) Choose the miva.dtd I downloaded from miva.com
3) After a short pause I get an error window saying "Error
while processing the DTD. The Error message is: x"

And that's it. I don't know if the problem is with the DTD
or with Quanta. So if possible I'd like a working DTEP. If
not, then a current DTD and I'll work with to try and get
Quanta to convert. But I don't want to spend hours trying
to get an old DTD working only to have to edit the
resulting file to include all the later features.

/Scott Mc

--- Roberto Buccino <[email protected]> wrote:

> Hi Scott
>
> I'm quanta user v3.0 on my RedHat 8.0 distro. I don't
> found that your DTD
> said.
>
>
> Roberto Buccino
> IT Developer
> ______________________________
> Alfa Centauro Custom System
> Copahue 3480 Melipal
> (02944) 44 11 88 / 15 60 3287
> ______________________________
> San Carlos de Bariloche - Rio Negro
> Patagonia - Argentina
> <A HREF ="http://www.paralelo42.com">http://www.paralelo42.com</A>
>
>
> El Martes 22 Febrero 2005 17:59, Scott McC escribió:
> > These two subjects go hand in hand so I'll post them
> > together.
> >
> > In some other coding projects I'm doing
> (non-MivaScript)
> > I've been using Quanta Plus and really enjoy it. Lots
> of
> > neat features like function folding, auto-complete,
> etc.
> >
> > I've searched high and low trying to find a DTEP file
> built
> > for MivaScript. Basically the file that contains all
> the
> > rule of how the language works. So if anyone else is
> using
> > Quanta for development and has a DTEP file I'd
> appreciate
> > if you could send me a copy.
> >
> > Barring that, Quanta does have in import feature. It
> will
> > import a DTD file and convert it into a DTEP file.
> > Searching Miva's website the only DTD I can find for
> the
> > language is for version 3.15. Positively ancient.
> Trying to
> > import that gives me an error. So, if I can't get the
> DTEP
> > right off the bat, does anyone know where to get a
> current
> > copy of the DTD for MivaScript 3.97 and 4.xx?
> >
> > /ScottMc
> >
>
>


__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Re: MvLOCKFILE (was: Adding fields to a database )



Hi Brian,

"Even though the manual refers to a "record lock" and a "file lock", I don't
believe there is any difference in what Empresa is doing."

I think there is a very important difference. A file lock applies to the
entire file (and it sets the .lck file), a record lock -at least in the
common definition- to a single record in a database. A record lock indeed
should prevent that two processes can update the -> same <- record at the
same time, however if this is really on the record level, then it should be
possible that two processes access two different records (almost)
simultaneously.

This obviously could create an index corruption if the new value of an added
or updated record depends on the database itself, for example a sequence or
serial value based on totrec, recno, last value + 1 etc.



It is also interesting to consider the statement in the reference manual
that it is not guaranteed which lock is executed first (unlike MvLockfile
requests which are sequential). So if you have the following calls, executed
in this order:

Process 1

MvASSIGN NAME=d.label VALUE=A
MvASSIGN NAME=d.myvalue VALUE={ value_from_function() + d.totrec}
MvADD

Process 2

MvASSIGN NAME=d.label VALUE=B
MvASSIGN NAME=d.myvalue VALUE={d.totrec}
MvADD


This could return 3 different results:

1.If the d.totrec of the database is for example 20, and
value_from_function() returns 10, the assignment will add 30 for d.myvalue.
The next process 1 will then writes 21 for d.myvalue, because totrec is now
one more. This is what one should expect.

Result:
Label MyVALUE
A 30
B 21

2. But what if value_from_function() takes a few moments to compute? In the
meantime, the CPU switches to Process 2 and assigns 20 (d.totrec), so
instead of the previously current value of 20 in process 1, now suddenly
this value is 21, and consequently the result of process 1 is 31:

Label MyVALUE
A 31
B 20

3.) Process 1 immediately takes the value 20 for d.totrec and then computes
the value_from_function(). During that computation process 2 assigns 20
(d.totrec) to the database. Then the CPU switches back to process 1 and adds
30 for the record:

Label MyVALUE
A 30
B 20

Why? Because of automatic record lock kicks in only later -at least that's
my guess- when the MvADD is called, and not during the MvASSIGN.

The example is a bit simplistic, but it illustrates a common problem in
concurrent programming (actually, that is exactly what non-deterministic
means). To prevent this we'd need to block write access completely before we
make the assignments, and so far the only way I know is through these ugly
MvLOCKFILE calls. Fortunately this is really only a problem if the value
depends on the database itself, like if we are using d.recnos or serial
values that are not "hard locked".

Markus






-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Brian (Burns Enterprises)
Sent: Mittwoch, 23. Februar 2005 23:55
To: Bill Matlock
Cc: Miva Users List
Subject: Re: [meu] Re: MvLOCKFILE (was: Adding fields to a database )

The second script calling does not need to use the MvLOCKFILE command.
Empresa is already performing this.

<snip>
"As indicated in the database sections, when you are adding (<MvADD>) or
updating (<MvUPDATE>) a database record, Miva automatically prevents the
same record from being updated by more than one process at the same time. No
other Miva Script process can access a record while it is locked. "

... And in the database section:

"Database and index files are locked at the record level automatically when
records are added or updated to prevent data corruption (that is, two
programs cannot update the same record at the same time). Entire files can
be locked using <MvLOCKFILE>."
</snip>

Even though the manual refers to a "record lock" and a "file lock", I don't
believe there is any difference in what Empresa is doing.

FYI: I know the caching bug affects databases (as per the original test
script <A HREF ="http://www.becoded.com/lockfile_test.txt)">http://www.becoded.com/lockfile_test.txt)</A>
I have not run these tests using read/write from a text file....so not sure
if this would be affected also.
(i.e. second script doesn't see line updated/added by first script)

-Brian
Burns Enterprises

On Wed, 23 Feb 2005 17:35:39 -0500, Bill Matlock <[email protected]>
wrote:
> Brian,
>
> > I think you're misunderstanding the issue.
> > If you lock a .dbf, then try to read/write to it from another
> > script, the script "will" wait until the lock is released.
>
> Even if the "other" script does not use a lock? This is where I'm
> fuzzy on the issue of MvLOCKFILE.
>
> Quote:
> "However, if the second process does not issue an <MvLOCKFILE> before
> attempting to access the file in question, there is nothing to prevent
> this access from taking place. For this reason, we refer to the action
> of <MvLOCKFILE> as a lock request, rather than a lock: it is a request
> to other Miva Script processes that they 'respect' the current
> process's request for exclusive access. Another way of looking at this
> is: <MvLOCKFILE> does not interact with any other Miva Script tag,
> only other <MvLOCKFILE>s."
>
> So, according to this, the 2nd script will need a lock in order for it
> to wait for the 1st script's lock to be released. If the 2nd script
> was just a simple read, it would still require a lock for it to wait
> for the first script's lock to be released? And if so, my sometimes
> unlogical mind tells me that in order to always have a lock "trully"
> lock a file, all database operations would need to be inside a lock as
> well, because - "<MvLOCKFILE> does not interact with any other Miva Script
tag, only other <MvLOCKFILE>s".
>
> Sorry if I'm confusing the issue, but this whole lockfile thing has
> always given me fits.
>
> Thanks,
> Bill M.
>

MvLOCKFILE (was: Adding fields to a database )



I saw several misunderstandings of the functionality of MvLOCK in this
thread.

The whole MvLOCK is simply a file mechanism signalling to another instance
of a script using the MvLOCK on the same file that it has to wait. It won't
prevent operations on the locked file from scripts or part of script not
enclosed in the corresponding MvLOCKFILE.

As for the nesting - you can certainly nest MvLOCK's, but it is intended for
locks on _different_ files. Using nested MvLOCK on the same file like in
Brian's (?) example makes no sense, and it is no wonder it does not work.

As for record blocking - I would love to believe that it works as described
in the documentation, but unfortunately have to very strongly doubt it.
Concurrently running Empresa processes run completely independently,
encapsulated in isolated CGI environments and memory space. On most OS'es, I
do not believe that these processes communicate (or even can efficiently
communicate) between themselves - so I do not think that one instance of
Empresa can easily tell ( = w/o writing to the disk) another one that a
specific database record is in use. That would be possible if there was a
common engine resident in the memory, but we know well that this is
unfortunately not the case (at least on Unix machines).

More likely (if at all) Empresa simply sets OS/BIOS based lock on the file
(on some OS'es it could theoretically lock a part of the file). Although it
may work sometimes, I fear (and from experience know) that it is really
neither reliable nor efficient. That could be really done only with a
dedicated memory resident database engine - and that's exactly the reason of
the latter tendency to integrate Empresa with Oracle, MySQL and other "real"
dB engines. It would be nonsense to reinvent the wheel and integrate the
functionality directly into Empresa.

Ivo
http://mivo.truxoft.com


-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of MvMarkus
Sent: Thursday, February 24, 2005 4:20 AM
To: 'Brian (Burns Enterprises)'; 'Bill Matlock'
Cc: 'Miva Users List'
Subject: RE: [meu] Re: MvLOCKFILE (was: Adding fields to a database )

Hi Brian,

"Even though the manual refers to a "record lock" and a "file lock", I don't
believe there is any difference in what Empresa is doing."

I think there is a very important difference. A file lock applies to the
entire file (and it sets the .lck file), a record lock -at least in the
common definition- to a single record in a database. A record lock indeed
should prevent that two processes can update the -> same <- record at the
same time, however if this is really on the record level, then it should be
possible that two processes access two different records (almost)
simultaneously.

This obviously could create an index corruption if the new value of an added
or updated record depends on the database itself, for example a sequence or
serial value based on totrec, recno, last value + 1 etc.



It is also interesting to consider the statement in the reference manual
that it is not guaranteed which lock is executed first (unlike MvLockfile
requests which are sequential). So if you have the following calls, executed
in this order:

Process 1

MvASSIGN NAME=d.label VALUE=A
MvASSIGN NAME=d.myvalue VALUE={ value_from_function() + d.totrec}
MvADD

Process 2

MvASSIGN NAME=d.label VALUE=B
MvASSIGN NAME=d.myvalue VALUE={d.totrec}
MvADD


This could return 3 different results:

1.If the d.totrec of the database is for example 20, and
value_from_function() returns 10, the assignment will add 30 for d.myvalue.
The next process 1 will then writes 21 for d.myvalue, because totrec is now
one more. This is what one should expect.

Result:
Label MyVALUE
A 30
B 21

2. But what if value_from_function() takes a few moments to compute? In the
meantime, the CPU switches to Process 2 and assigns 20 (d.totrec), so
instead of the previously current value of 20 in process 1, now suddenly
this value is 21, and consequently the result of process 1 is 31:

Label MyVALUE
A 31
B 20

3.) Process 1 immediately takes the value 20 for d.totrec and then computes
the value_from_function(). During that computation process 2 assigns 20
(d.totrec) to the database. Then the CPU switches back to process 1 and adds
30 for the record:

Label MyVALUE
A 30
B 20

Why? Because of automatic record lock kicks in only later -at least that's
my guess- when the MvADD is called, and not during the MvASSIGN.

The example is a bit simplistic, but it illustrates a common problem in
concurrent programming (actually, that is exactly what non-deterministic
means). To prevent this we'd need to block write access completely before we
make the assignments, and so far the only way I know is through these ugly
MvLOCKFILE calls. Fortunately this is really only a problem if the value
depends on the database itself, like if we are using d.recnos or serial
values that are not "hard locked".

Markus






-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Brian (Burns Enterprises)
Sent: Mittwoch, 23. Februar 2005 23:55
To: Bill Matlock
Cc: Miva Users List
Subject: Re: [meu] Re: MvLOCKFILE (was: Adding fields to a database )

The second script calling does not need to use the MvLOCKFILE command.
Empresa is already performing this.

<snip>
"As indicated in the database sections, when you are adding (<MvADD>) or
updating (<MvUPDATE>) a database record, Miva automatically prevents the
same record from being updated by more than one process at the same time. No
other Miva Script process can access a record while it is locked. "

... And in the database section:

"Database and index files are locked at the record level automatically when
records are added or updated to prevent data corruption (that is, two
programs cannot update the same record at the same time). Entire files can
be locked using <MvLOCKFILE>."
</snip>

Even though the manual refers to a "record lock" and a "file lock", I don't
believe there is any difference in what Empresa is doing.

FYI: I know the caching bug affects databases (as per the original test
script <A HREF ="http://www.becoded.com/lockfile_test.txt)">http://www.becoded.com/lockfile_test.txt)</A>
I have not run these tests using read/write from a text file....so not sure
if this would be affected also.
(i.e. second script doesn't see line updated/added by first script)

-Brian
Burns Enterprises

On Wed, 23 Feb 2005 17:35:39 -0500, Bill Matlock <[email protected]>
wrote:
> Brian,
>
> > I think you're misunderstanding the issue.
> > If you lock a .dbf, then try to read/write to it from another
> > script, the script "will" wait until the lock is released.
>
> Even if the "other" script does not use a lock? This is where I'm
> fuzzy on the issue of MvLOCKFILE.
>
> Quote:
> "However, if the second process does not issue an <MvLOCKFILE> before
> attempting to access the file in question, there is nothing to prevent
> this access from taking place. For this reason, we refer to the action
> of <MvLOCKFILE> as a lock request, rather than a lock: it is a request
> to other Miva Script processes that they 'respect' the current
> process's request for exclusive access. Another way of looking at this
> is: <MvLOCKFILE> does not interact with any other Miva Script tag,
> only other <MvLOCKFILE>s."
>
> So, according to this, the 2nd script will need a lock in order for it
> to wait for the 1st script's lock to be released. If the 2nd script
> was just a simple read, it would still require a lock for it to wait
> for the first script's lock to be released? And if so, my sometimes
> unlogical mind tells me that in order to always have a lock "trully"
> lock a file, all database operations would need to be inside a lock as
> well, because - "<MvLOCKFILE> does not interact with any other Miva Script
tag, only other <MvLOCKFILE>s".
>
> Sorry if I'm confusing the issue, but this whole lockfile thing has
> always given me fits.
>
> Thanks,
> Bill M.
>

MvLOCKFILE (was: Adding fields to a database )



Hi Ivo,

It's about time that you join this discussion!

The point that you mention at the end of your text is particulary
interesting: How (if) can Empresa create a record lock which, according to
the docs also locks the open indices, if different processes do not
communicate with each other?

My guess is that there is probably no such mechanism in Empresa, but instead
the "record locking mechanism" in the docs refers to nothing else but an
exclusive write access of the OS to the address (space) of the record on the
disk.

(Alternatively, is it possible that dbIIIs have an internal
"record-is-updated-right-now-flag-do-not-touch-it flag" similar to d.deleted
?)

Whatever it is, this does not help with finding a solution to the problem
that I described in my previous post about preventing the database (or
index) to change in the time between the assignment of the new database
values and the beginning of the database transaction.

Therefore, if the new value depends on other values in the database (or the
number/position of records in the database or index), the only remaining
available tool is MvLOCKFILE. But to get it to work properly, all functions
that could possibly write to that table and index must explicitely request
it. Or not? A horrible idea, in my opionion, and a clear contradition to the
recommandations in the docs!



"It would be nonsense to reinvent the wheel and integrate the functionality
directly into Empresa."

I don't think that this is really the objective here. The whole discussion
aims to

1.) Finding a secure way to protect tables from index corruptions during
concurrent updates or adds. We all know that this is an existing problem in
Miva, especially as the error handling provides warnings of such a
corruption, but no integrated protections/recovery. When you have a
duplicate error in the index, the incorrect value is still (first) written
into the table. This is, in my opionion, a real design flaw in the dbIIIs or
Empresa.

2.) Gaining better understanding over the different locking mechanisms,
their functionality, applications, reliablity, performance costs, to be
ultimately able to write scripts that are robust and efficient, even under
heavy load.

Of course we can always say "forget dbIIIs, I use a SQL db". But that's no
fun.

Markus













-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Ivo Truxa
Sent: Donnerstag, 24. Februar 2005 13:52
To: 'Miva Users List'
Subject: RE: [meu] MvLOCKFILE (was: Adding fields to a database )

I saw several misunderstandings of the functionality of MvLOCK in this
thread.

The whole MvLOCK is simply a file mechanism signalling to another instance
of a script using the MvLOCK on the same file that it has to wait. It won't
prevent operations on the locked file from scripts or part of script not
enclosed in the corresponding MvLOCKFILE.

As for the nesting - you can certainly nest MvLOCK's, but it is intended for
locks on _different_ files. Using nested MvLOCK on the same file like in
Brian's (?) example makes no sense, and it is no wonder it does not work.

As for record blocking - I would love to believe that it works as described
in the documentation, but unfortunately have to very strongly doubt it.
Concurrently running Empresa processes run completely independently,
encapsulated in isolated CGI environments and memory space. On most OS'es, I
do not believe that these processes communicate (or even can efficiently
communicate) between themselves - so I do not think that one instance of
Empresa can easily tell ( = w/o writing to the disk) another one that a
specific database record is in use. That would be possible if there was a
common engine resident in the memory, but we know well that this is
unfortunately not the case (at least on Unix machines).

More likely (if at all) Empresa simply sets OS/BIOS based lock on the file
(on some OS'es it could theoretically lock a part of the file). Although it
may work sometimes, I fear (and from experience know) that it is really
neither reliable nor efficient. That could be really done only with a
dedicated memory resident database engine - and that's exactly the reason of
the latter tendency to integrate Empresa with Oracle, MySQL and other "real"
dB engines. It would be nonsense to reinvent the wheel and integrate the
functionality directly into Empresa.

Ivo
http://mivo.truxoft.com


-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of MvMarkus
Sent: Thursday, February 24, 2005 4:20 AM
To: 'Brian (Burns Enterprises)'; 'Bill Matlock'
Cc: 'Miva Users List'
Subject: RE: [meu] Re: MvLOCKFILE (was: Adding fields to a database )

Hi Brian,

"Even though the manual refers to a "record lock" and a "file lock", I don't
believe there is any difference in what Empresa is doing."

I think there is a very important difference. A file lock applies to the
entire file (and it sets the .lck file), a record lock -at least in the
common definition- to a single record in a database. A record lock indeed
should prevent that two processes can update the -> same <- record at the
same time, however if this is really on the record level, then it should be
possible that two processes access two different records (almost)
simultaneously.

This obviously could create an index corruption if the new value of an added
or updated record depends on the database itself, for example a sequence or
serial value based on totrec, recno, last value + 1 etc.



It is also interesting to consider the statement in the reference manual
that it is not guaranteed which lock is executed first (unlike MvLockfile
requests which are sequential). So if you have the following calls, executed
in this order:

Process 1

MvASSIGN NAME=d.label VALUE=A
MvASSIGN NAME=d.myvalue VALUE={ value_from_function() + d.totrec} MvADD

Process 2

MvASSIGN NAME=d.label VALUE=B
MvASSIGN NAME=d.myvalue VALUE={d.totrec} MvADD


This could return 3 different results:

1.If the d.totrec of the database is for example 20, and
value_from_function() returns 10, the assignment will add 30 for d.myvalue.
The next process 1 will then writes 21 for d.myvalue, because totrec is now
one more. This is what one should expect.

Result:
Label MyVALUE
A 30
B 21

2. But what if value_from_function() takes a few moments to compute? In the
meantime, the CPU switches to Process 2 and assigns 20 (d.totrec), so
instead of the previously current value of 20 in process 1, now suddenly
this value is 21, and consequently the result of process 1 is 31:

Label MyVALUE
A 31
B 20

3.) Process 1 immediately takes the value 20 for d.totrec and then computes
the value_from_function(). During that computation process 2 assigns 20
(d.totrec) to the database. Then the CPU switches back to process 1 and adds
30 for the record:

Label MyVALUE
A 30
B 20

Why? Because of automatic record lock kicks in only later -at least that's
my guess- when the MvADD is called, and not during the MvASSIGN.

The example is a bit simplistic, but it illustrates a common problem in
concurrent programming (actually, that is exactly what non-deterministic
means). To prevent this we'd need to block write access completely before we
make the assignments, and so far the only way I know is through these ugly
MvLOCKFILE calls. Fortunately this is really only a problem if the value
depends on the database itself, like if we are using d.recnos or serial
values that are not "hard locked".

Markus






-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of Brian (Burns Enterprises)
Sent: Mittwoch, 23. Februar 2005 23:55
To: Bill Matlock
Cc: Miva Users List
Subject: Re: [meu] Re: MvLOCKFILE (was: Adding fields to a database )

The second script calling does not need to use the MvLOCKFILE command.
Empresa is already performing this.

<snip>
"As indicated in the database sections, when you are adding (<MvADD>) or
updating (<MvUPDATE>) a database record, Miva automatically prevents the
same record from being updated by more than one process at the same time. No
other Miva Script process can access a record while it is locked. "

... And in the database section:

"Database and index files are locked at the record level automatically when
records are added or updated to prevent data corruption (that is, two
programs cannot update the same record at the same time). Entire files can
be locked using <MvLOCKFILE>."
</snip>

Even though the manual refers to a "record lock" and a "file lock", I don't
believe there is any difference in what Empresa is doing.

FYI: I know the caching bug affects databases (as per the original test
script <A HREF ="http://www.becoded.com/lockfile_test.txt)">http://www.becoded.com/lockfile_test.txt)</A>
I have not run these tests using read/write from a text file....so not sure
if this would be affected also.
(i.e. second script doesn't see line updated/added by first script)

-Brian
Burns Enterprises

On Wed, 23 Feb 2005 17:35:39 -0500, Bill Matlock <[email protected]>
wrote:
> Brian,
>
> > I think you're misunderstanding the issue.
> > If you lock a .dbf, then try to read/write to it from another
> > script, the script "will" wait until the lock is released.
>
> Even if the "other" script does not use a lock? This is where I'm
> fuzzy on the issue of MvLOCKFILE.
>
> Quote:
> "However, if the second process does not issue an <MvLOCKFILE> before
> attempting to access the file in question, there is nothing to prevent
> this access from taking place. For this reason, we refer to the action
> of <MvLOCKFILE> as a lock request, rather than a lock: it is a request
> to other Miva Script processes that they 'respect' the current
> process's request for exclusive access. Another way of looking at this
> is: <MvLOCKFILE> does not interact with any other Miva Script tag,
> only other <MvLOCKFILE>s."
>
> So, according to this, the 2nd script will need a lock in order for it
> to wait for the 1st script's lock to be released. If the 2nd script
> was just a simple read, it would still require a lock for it to wait
> for the first script's lock to be released? And if so, my sometimes
> unlogical mind tells me that in order to always have a lock "trully"
> lock a file, all database operations would need to be inside a lock as
> well, because - "<MvLOCKFILE> does not interact with any other Miva Script
tag, only other <MvLOCKFILE>s".
>
> Sorry if I'm confusing the issue, but this whole lockfile thing has
> always given me fits.
>
> Thanks,
> Bill M.
>

MvLOCKFILE (was: Adding fields to a database )



Yes, that was exactly the point. I do not think that the record locking
works, ever worked, or will ever work perfectly with the native xBase Miva
Script databases. Even if there were a hidden field in the dBase file (and I
am pretty sure there is none), it would not solve the problem - the time to
turn the flag would be practically identical to the time writing the entire
record, hence still letting a time window open for a possible conflict, or
considerably slowing the entire system. Only a memory resident solution, not
a file based one, can help. Well, at current speeds of computers, the
probability of a conflict is relatively low, and it is a good trade for the
simplicity of a dBase system, but I am afraid that without a full-bloated
database engine you won't get any perfection.

Ivo Truxa

| http://miva.truxoft.com
| Advanced Miva Merchant modules



-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf
Of MvMarkus
Sent: Thursday, February 24, 2005 9:24 PM
To: 'Ivo Truxa'; 'Miva Users List'
Subject: RE: [meu] MvLOCKFILE (was: Adding fields to a database )

Hi Ivo,

It's about time that you join this discussion!

The point that you mention at the end of your text is particulary
interesting: How (if) can Empresa create a record lock which, according to
the docs also locks the open indices, if different processes do not
communicate with each other?

My guess is that there is probably no such mechanism in Empresa, but instead
the "record locking mechanism" in the docs refers to nothing else but an
exclusive write access of the OS to the address (space) of the record on the
disk.

(Alternatively, is it possible that dbIIIs have an internal
"record-is-updated-right-now-flag-do-not-touch-it flag" similar to d.deleted
?)

Whatever it is, this does not help with finding a solution to the problem
that I described in my previous post about preventing the database (or
index) to change in the time between the assignment of the new database
values and the beginning of the database transaction.

Therefore, if the new value depends on other values in the database (or the
number/position of records in the database or index), the only remaining
available tool is MvLOCKFILE. But to get it to work properly, all functions
that could possibly write to that table and index must explicitely request
it. Or not? A horrible idea, in my opionion, and a clear contradition to the
recommandations in the docs!



"It would be nonsense to reinvent the wheel and integrate the functionality
directly into Empresa."

I don't think that this is really the objective here. The whole discussion
aims to

1.) Finding a secure way to protect tables from index corruptions during
concurrent updates or adds. We all know that this is an existing problem in
Miva, especially as the error handling provides warnings of such a
corruption, but no integrated protections/recovery. When you have a
duplicate error in the index, the incorrect value is still (first) written
into the table. This is, in my opionion, a real design flaw in the dbIIIs or
Empresa.

2.) Gaining better understanding over the different locking mechanisms,
their functionality, applications, reliablity, performance costs, to be
ultimately able to write scripts that are robust and efficient, even under
heavy load.

Of course we can always say "forget dbIIIs, I use a SQL db". But that's no
fun.

Markus

MvLOCKFILE (was: Adding fields to a database )



I just wanted to interject a few thoughts.

First to the issue of the MvLOCKFILE tag and whether it is
useful, and whether it works. I think that it is certainly
useful, and certainly works to the extent intended, and
programming would be hell without it. I know because I
programmed in the past long ago without it and it was hell,
and now with it that particular hell is gone.

In my opinion the single most useful thing that MvLOCKFILE
does in most cases is lock the code you use to generate
unique keys. This way when you are building a database
structure you may have as complex a set of relations between
tables as you want, all keyed using unique keys, and if you
are careful to make sure that all requests for a new key get
the key generated from the MVLOCKFILE protected code you
will be very safe. The second most important thing is the
closely related issue of wrapping update code so that unique
index keys that are input by end users, such as codes,
logins, etc. are all wrapped as well, so that when your
application does a seek into the database to see if the code
already exists, and then not finding it allows it to be
added, you will know that no other request was looking up to
see if that same code existed before trying to add it
because the request to find it and add it are all wrapped in
the MVLOCKFILE.

Also don't forget the value of locking plain text files.
Using MvEXPORT and MvIMPORT with a filter applied and
"packing" the text file inside an MVLOCKFILE can actually
produce very useful results.

And one more thing, don't forget that the MvLOCKFILE isn't
tied to a specific database file, the FILE you specify is an
arbitrary string that becomes part of the name of the lock
file that is crated on the server, and can be dynamic so
that for example if you wanted to throttle back on the
number of resource requests from a single IP address or
based on a cookie, you could wrap an MvLOCKFILE around the
code that outputs whatever you want to throttle and use the
IP address or cookie as the FILE and thus if an address was
sending 20 requests for the type of resource all at near the
same time you would make them get them serially rather than
all at once.

As for the issue of SQL support in Miva script and how
things will be in the future, we all shouldn't expect that
just having Miva support a database system, like MySQL for
example, is going to solve all problems. MySQL databases get
corrupted, conflicts occur, network access to the database
server can fail, application level bugs still allow bad data
to get entered or wrong records to be cross referenced, etc.

And worse than that there are going to end up being lots of
portability issues that arise. Miva script programmers have
been spared such issues almost completely. Even really old
version of interpreted Miva script and the newest versions
of the 4.x Miva script under the VM have all had database
code that has been seamlessly transferable in most cases
from one hosted environment to another with no problems even
when the version of the Engine changed from environment to
environment -- with the specific exception of the changes in
the way date fields were handled -- and portability problems
from server to server both running the same version of
Empresa are virtually unheard of.

This ease of portability is NOT going to be the case going
foreword. I predict a whole series of issues that
programmers will have to face related to their scripts
behaving differently on one system than they do on another,
and that the coding styles will have to change to
accommodate the portability issues.

Of course we will be gaining a lot once the various versions
of SQL support are available to us, and it is thus worth
the pain that will come with it. Or at least I hope it will
be :)

Thankfully we heard nothing at the conference that leads us
to believe that Miva has any plans of dropping dBase support
so we can hopefully still rely on that for all the things
that don't require SQL.

That's enough of my rambling thoughts for a while.

- Jeff Huber
President 4TheBest eCommerce Solutions
http://4TheBest.com
[email protected]
Office: 760-742-1469
Cell: 760-445-8454

MvLOCKFILE (was: Adding fields to a database )



I agree with Jeff
<snip>
In my opinion the single most useful thing that MvLOCKFILE
does in most cases is lock the code you use to generate
unique keys.
</snip>

This, after all, was the whole reason behind my original post a year ago.
And as of Merchant 4.13, Merchant was still incorrectly generating unique keys.

-Brian
Burns Enterprises

On Thu, 24 Feb 2005 20:05:57 -0800, Jeff Huber - Listmail
<[email protected]> wrote:
> I just wanted to interject a few thoughts.
>
> First to the issue of the MvLOCKFILE tag and whether it is
> useful, and whether it works. I think that it is certainly
> useful, and certainly works to the extent intended, and
> programming would be hell without it. I know because I
> programmed in the past long ago without it and it was hell,
> and now with it that particular hell is gone.
>
> In my opinion the single most useful thing that MvLOCKFILE
> does in most cases is lock the code you use to generate
> unique keys. This way when you are building a database
> structure you may have as complex a set of relations between
> tables as you want, all keyed using unique keys, and if you
> are careful to make sure that all requests for a new key get
> the key generated from the MVLOCKFILE protected code you
> will be very safe. The second most important thing is the
> closely related issue of wrapping update code so that unique
> index keys that are input by end users, such as codes,
> logins, etc. are all wrapped as well, so that when your
> application does a seek into the database to see if the code
> already exists, and then not finding it allows it to be
> added, you will know that no other request was looking up to
> see if that same code existed before trying to add it
> because the request to find it and add it are all wrapped in
> the MVLOCKFILE.
>
> Also don't forget the value of locking plain text files.
> Using MvEXPORT and MvIMPORT with a filter applied and
> "packing" the text file inside an MVLOCKFILE can actually
> produce very useful results.
>
> And one more thing, don't forget that the MvLOCKFILE isn't
> tied to a specific database file, the FILE you specify is an
> arbitrary string that becomes part of the name of the lock
> file that is crated on the server, and can be dynamic so
> that for example if you wanted to throttle back on the
> number of resource requests from a single IP address or
> based on a cookie, you could wrap an MvLOCKFILE around the
> code that outputs whatever you want to throttle and use the
> IP address or cookie as the FILE and thus if an address was
> sending 20 requests for the type of resource all at near the
> same time you would make them get them serially rather than
> all at once.
>
> As for the issue of SQL support in Miva script and how
> things will be in the future, we all shouldn't expect that
> just having Miva support a database system, like MySQL for
> example, is going to solve all problems. MySQL databases get
> corrupted, conflicts occur, network access to the database
> server can fail, application level bugs still allow bad data
> to get entered or wrong records to be cross referenced, etc.
>
> And worse than that there are going to end up being lots of
> portability issues that arise. Miva script programmers have
> been spared such issues almost completely. Even really old
> version of interpreted Miva script and the newest versions
> of the 4.x Miva script under the VM have all had database
> code that has been seamlessly transferable in most cases
> from one hosted environment to another with no problems even
> when the version of the Engine changed from environment to
> environment -- with the specific exception of the changes in
> the way date fields were handled -- and portability problems
> from server to server both running the same version of
> Empresa are virtually unheard of.
>
> This ease of portability is NOT going to be the case going
> foreword. I predict a whole series of issues that
> programmers will have to face related to their scripts
> behaving differently on one system than they do on another,
> and that the coding styles will have to change to
> accommodate the portability issues.
>
> Of course we will be gaining a lot once the various versions
> of SQL support are available to us, and it is thus worth
> the pain that will come with it. Or at least I hope it will
> be :)
>
> Thankfully we heard nothing at the conference that leads us
> to believe that Miva has any plans of dropping dBase support
> so we can hopefully still rely on that for all the things
> that don't require SQL.
>
> That's enough of my rambling thoughts for a while.
>
> - Jeff Huber
> President 4TheBest eCommerce Solutions
> http://4TheBest.com
> [email protected]
> Office: 760-742-1469
> Cell: 760-445-8454
>

MvLOCKFILE (was: Adding fields to a database )



On Thu, 24 Feb 2005 20:05:57 -0800, Jeff Huber - Listmail
<[email protected]> wrote:

> Also don't forget the value of locking plain text files.
> Using MvEXPORT and MvIMPORT with a filter applied and
> "packing" the text file inside an MVLOCKFILE can actually
> produce very useful results.

Yep, doing that in an app now. Not only for files that have filters,
but also good for plain text files that you need to "protect".

> And one more thing, don't forget that the MvLOCKFILE isn't
> tied to a specific database file, the FILE you specify is an
> arbitrary string that becomes part of the name of the lock
> file that is crated on the server, and can be dynamic so
> that for example if you wanted to throttle back on the
> number of resource requests from a single IP address or
> based on a cookie, you could wrap an MvLOCKFILE around the
> code that outputs whatever you want to throttle and use the
> IP address or cookie as the FILE and thus if an address was
> sending 20 requests for the type of resource all at near the
> same time you would make them get them serially rather than
> all at once.

Bravo (*golf clap*)! For this one, we'll have to see if we can add
you to the "Miva Player of the Week" page:
http://miva.osu.edu/pow.htm

which would entitle you to add this nifty Miva logo to your site:
http://miva.osu.edu/photos/LOmiva.jpg.jpg

> As for the issue of SQL support in Miva script and how
> things will be in the future, we all shouldn't expect that
> just having Miva support a database system, like MySQL for
> example, is going to solve all problems. MySQL databases get
> corrupted, conflicts occur, network access to the database
> server can fail, application level bugs still allow bad data
> to get entered or wrong records to be cross referenced, etc.

Agreed. MySQL helps, but it's no magic bullet.

> And worse than that there are going to end up being lots of
> portability issues that arise. Miva script programmers have
> been spared such issues almost completely. Even really old
> version of interpreted Miva script and the newest versions
> of the 4.x Miva script under the VM have all had database
> code that has been seamlessly transferable in most cases
> from one hosted environment to another with no problems even
> when the version of the Engine changed from environment to
> environment -- with the specific exception of the changes in
> the way date fields were handled -- and portability problems
> from server to server both running the same version of
> Empresa are virtually unheard of.
>
> This ease of portability is NOT going to be the case going
> foreword. I predict a whole series of issues that
> programmers will have to face related to their scripts
> behaving differently on one system than they do on another,
> and that the coding styles will have to change to
> accommodate the portability issues.

Agreed. Not all of us have Apache/MySQL installed, but I imagine
there will be a flurry of threads related to them soon -- btw, Apache
is a nice addition anyway, as you can use mod_rewrite to help you with
local links to .mv and .mvc files (it can add the appropriate ports
for you)..

> Of course we will be gaining a lot once the various versions
> of SQL support are available to us, and it is thus worth
> the pain that will come with it. Or at least I hope it will
> be :)
>
> Thankfully we heard nothing at the conference that leads us
> to believe that Miva has any plans of dropping dBase support
> so we can hopefully still rely on that for all the things
> that don't require SQL.
>
> That's enough of my rambling thoughts for a while.

I'm hoping you're right about both of those... wider range of SQL
support, and continued xBase support.

--
Bill Guindon (aka aGorilla)

MvLOCKFILE (was: Adding fields to a database )



On Thu, 24 Feb 2005 23:56:21 -0500, Bill Guindon <[email protected]> wrote:

> > if you wanted to throttle back on the
> > number of resource requests from a single IP address or
> > based on a cookie, you could wrap an MvLOCKFILE around the
> > code that outputs whatever you want to throttle and use the
> > IP address or cookie as the FILE and thus if an address was
> > sending 20 requests for the type of resource all at near the
> > same time you would make them get them serially rather than
> > all at once.
>
> Bravo (*golf clap*)! For this one, we'll have to see if we can add
> you to the "Miva Player of the Week" page:
> http://miva.osu.edu/pow.htm
>
> which would entitle you to add this nifty Miva logo to your site:
> http://miva.osu.edu/photos/LOmiva.jpg.jpg

It was pointed out to me (off list) that my comments above might
appear to be sarcasm. For the record, it is NOT sarcasm. I was just
looking for a fun way to point out those odd links I'd come across --
and Jeff's tip was worthy of a "Miva Player of the Week" award.

I'm truly impressed by such a novel use of the tag. I doubt that I
would have ever thought to use a 'lock' on an IP or cookie (or
anything other than a file for that matter), and it certainly does
have it's uses.

Thanks Jeff, for sharing it with us.

--
Bill Guindon (aka aGorilla)

MvLOCKFILE (was: Adding fields to a database )



Bill,

For the record I took your message in the spirit you intended it.

- Jeff Huber
President 4TheBest eCommerce Solutions
http://4TheBest.com
[email protected]
Office: 760-742-1469
Cell: 760-445-8454
=20


-----Original Message-----
From: Bill Guindon [mailto:[email protected]]=20
Sent: Thursday, February 24, 2005 10:38 PM
To: Jeff Huber - Listmail
Cc: Miva Userlist
Subject: Re: [meu] MvLOCKFILE (was: Adding fields to a database )


On Thu, 24 Feb 2005 23:56:21 -0500, Bill Guindon
<[email protected]> wrote:

> > if you wanted to throttle back on the
> > number of resource requests from a single IP address or
> > based on a cookie, you could wrap an MvLOCKFILE around the
> > code that outputs whatever you want to throttle and use the
> > IP address or cookie as the FILE and thus if an address was
> > sending 20 requests for the type of resource all at near the
> > same time you would make them get them serially rather than
> > all at once.
>=20
> Bravo (*golf clap*)! For this one, we'll have to see if we can
add
> you to the "Miva Player of the Week" page:
> http://miva.osu.edu/pow.htm
>=20
> which would entitle you to add this nifty Miva logo to your
site:
> http://miva.osu.edu/photos/LOmiva.jpg.jpg

It was pointed out to me (off list) that my comments above might
appear to be sarcasm. For the record, it is NOT sarcasm. I was
just
looking for a fun way to point out those odd links I'd come
across --
and Jeff's tip was worthy of a "Miva Player of the Week" award.

I'm truly impressed by such a novel use of the tag. I doubt that
I
would have ever thought to use a 'lock' on an IP or cookie (or
anything other than a file for that matter), and it certainly
does
have it's uses.

Thanks Jeff, for sharing it with us.

--=20
Bill Guindon (aka aGorilla)

MvLOCKFILE (was: Adding fields to a database )



Just for fun.. for the "miva around the world" episode .. here are some miva
non-related websites that I found with google..

<A HREF ="http://www.miva.org">http://www.miva.org</A>
<A HREF ="http://www.miva-vball.org/">http://www.miva-vball.org/</A>
<A HREF ="http://www.survive-miva.org/">http://www.survive-miva.org/</A>
<A HREF ="http://www.miva-uv.sk/index_en.htm">http://www.miva-uv.sk/index_en.htm</A>

<A HREF ="http://www.miva.nl/">http://www.miva.nl/</A>
<A HREF ="http://www.miva-vakanties.nl/">http://www.miva-vakanties.nl/</A>
<A HREF ="http://www.miva.hu">http://www.miva.hu</A>

<A HREF ="http://www.miva.cz/">http://www.miva.cz/</A>
<A HREF ="http://www.miva-bohemia.cz/">http://www.miva-bohemia.cz/</A>
<A HREF ="http://www.miva-wt.cz/">http://www.miva-wt.cz/</A>

<A HREF ="http://www.miva.ch">http://www.miva.ch</A>
<A HREF ="http://www.miva.sk/">http://www.miva.sk/</A>
<A HREF ="http://www.miva.pl/">http://www.miva.pl/</A>

<A HREF ="http://www.miva.at/MIVA_Start/MIVA_frame.html">http://www.miva.at/MIVA_Start/MIVA_frame.html</A>

<A HREF ="http://www.miva.it/">http://www.miva.it/</A>
http://gral.ip.rm.cnr.it/software/Miva.htm

Claudiu

-----Original Message-----
From: [email protected] [mailto:[email protected]]On
Behalf Of Bill Guindon
Sent: vendredi 25 fevrier 2005 07:38
To: Jeff Huber - Listmail
Cc: Miva Userlist
Subject: Re: [meu] MvLOCKFILE (was: Adding fields to a database )


On Thu, 24 Feb 2005 23:56:21 -0500, Bill Guindon <[email protected]> wrote:

> > if you wanted to throttle back on the
> > number of resource requests from a single IP address or
> > based on a cookie, you could wrap an MvLOCKFILE around the
> > code that outputs whatever you want to throttle and use the
> > IP address or cookie as the FILE and thus if an address was
> > sending 20 requests for the type of resource all at near the
> > same time you would make them get them serially rather than
> > all at once.
>
> Bravo (*golf clap*)! For this one, we'll have to see if we can add
> you to the "Miva Player of the Week" page:
> http://miva.osu.edu/pow.htm
>
> which would entitle you to add this nifty Miva logo to your site:
> http://miva.osu.edu/photos/LOmiva.jpg.jpg

It was pointed out to me (off list) that my comments above might
appear to be sarcasm. For the record, it is NOT sarcasm. I was just
looking for a fun way to point out those odd links I'd come across --
and Jeff's tip was worthy of a "Miva Player of the Week" award.

I'm truly impressed by such a novel use of the tag. I doubt that I
would have ever thought to use a 'lock' on an IP or cookie (or
anything other than a file for that matter), and it certainly does
have it's uses.

Thanks Jeff, for sharing it with us.

--
Bill Guindon (aka aGorilla)