List of MivaScript builtin functions and discussion thereof

[wmgilligan]

Re: List of MivaScript builtin functions and discussion thereof

Just wanted to say thanks to burch. This is a great resource.
bill

[burch]

Re: List of MivaScript builtin functions and discussion thereof

Bravo! I haven't seen documentation on these "_mem" functions, but I do have docs on the older versions that used files, so I think I can figure out everything I need now. Mainly I needed to know what values to use for "bits" and "e."

bits is the size of the key, in bits. We use 1024-bit keys in Miva Merchant right now. Other meaningful values would be 2048 or 4096. I believe keys up to about 800 bits have been cracked in the past.

e is the RSA public key exponent. If you're a math geek you can see what it does at http://en.wikipedia.org/wiki/RSA Common values are 3, 17, and 65537. There are both security and performance ramifications to the value you choose for e. 65537 is probably the safest general purpose value.

Is there a release note or something that documents the _mem encryption functions?

The _mem functions just operate on variables for input and output instead of writing (or reading) the data from a file.

[Kent Multer]

Re: List of MivaScript builtin functions and discussion thereof

Bravo! I haven't seen documentation on these "_mem" functions, but I do have docs on the older versions that used files, so I think I can figure out everything I need now. Mainly I needed to know what values to use for "bits" and "e."

Is there a release note or something that documents the _mem encryption functions?

Thanks again --

[burch]

Re: List of MivaScript builtin functions and discussion thereof

Here's Encryption_Key_Create().

<MvFUNCTION NAME = "Encryption_Key_Create" PARAMETERS = "encryption var, passphrase" STANDARDOUTPUTLEVEL = "">
    <MvTRANSACT NAME = "Merchant">
    <MvIF EXPR = "{ NOT rsa_generate_keypair_mem( l.encryption:pub_key, l.encryption:prv_key, 1024, 17, l.passphrase ) }">
        <MvROLLBACK NAME = "Merchant">
        <MvFUNCTIONRETURN VALUE = "{ [ g.Library_Filename_Utilities ].Error( 'MER-CRY-00001', 'Unable to generate encryption key pair: ' $ crypto_last_error() ) }">
    </MvIF>
    <MvIF EXPR = "{ NOT [ g.Library_Filename_DB ].Encryption_Insert( l.encryption ) }">
        <MvROLLBACK NAME = "Merchant">
        <MvFUNCTIONRETURN VALUE = 0>
    </MvIF>
    <MvCOMMIT NAME = "Merchant">
    <MvFUNCTIONRETURN VALUE = 1>
</MvFUNCTION>

For the record this is the last meaningful function in lib/crypto.mv. The only other one is Encryption_Available() which just returns s.miva_sslavailable.

[Kent Multer]

Re: List of MivaScript builtin functions and discussion thereof

Wow, Jon, this is great! I'm starting to feel like Aladdin here ... do I get another wish? :)

I was wrong before: those two functions don't quite give us the whole picture. We also need to see the part of admin.mv that generates key files when the user enters a new passphrase. Most of that may be in crypto.mv; if not, I hope you can publish the relevant snippet from admin.

Thanks again --

[burch]

Re: List of MivaScript builtin functions and discussion thereof

The notes you gave on Merchant's built-in encryption functions are excellent. If you want to complete the doc package with one click, you can just publish the source code for those two functions. That'll tell the developers everything we need to know about how to use the Miva encryption functions in a real-world way.

Here you go. We'll probably include lib/crypto.mv in the update to the LSK as well.

<MvFUNCTION NAME = "Encrypt_Payment" PARAMETERS = "crypt_id, secure_data var, payment_secure var" STANDARDOUTPUTLEVEL = "">
    <MvIF EXPR = "{ ( len( l.secure_data ) EQ 0 ) OR ( NOT l.crypt_id ) }">
        <MvASSIGN NAME = "l.payment_secure:id"      VALUE = 0>
        <MvASSIGN NAME = "l.payment_secure:key"     VALUE = "">
        <MvASSIGN NAME = "l.payment_secure:data"    VALUE = "{ l.secure_data }">
        <MvFUNCTIONRETURN VALUE = 1>
    </MvIF>
    <MvIF EXPR = "{ NOT [ g.Library_Filename_DB ].Encryption_Load_ID( l.crypt_id, l.encryption ) }">
        <MvFUNCTIONRETURN VALUE = 0>
    </MvIF>
    <MvASSIGN NAME = "l.payment_secure:id" VALUE = "{ l.crypt_id }">
    <MvIF EXPR = "{ NOT rsa_load_publickey_mem( l.encryption:pub_key, l.rsa ) }">
        <MvFUNCTIONRETURN VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00002', 'Unable to load public key file \'' $ l.pubkey_file $ '\': ' $ crypto_last_error() ) }">
    </MvIF>
    <MvASSIGN NAME = "l.ok"                 VALUE = 1>
    <MvASSIGN NAME = "l.payment_key"        VALUE = "{ crypto_rand_bytes( 16 ) }">
    <MvIF EXPR = "{ l.ok AND NOT bf_encrypt( l.payment_key, l.secure_data, l.secure_data_encrypted ) }">
        <MvASSIGN NAME = "l.ok" VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00003', 'Unable to encrypt payment data: ' $ crypto_last_error() ) }">
    <MvELSE>
        <MvASSIGN NAME = "l.payment_secure:data"    VALUE = "{ crypto_base64_encode( l.secure_data_encrypted ) }">
    </MvIF>
    <MvIF EXPR = "{ l.ok AND NOT rsa_public_encrypt( l.rsa, l.payment_key, l.key_encrypted ) }">
        <MvASSIGN NAME = "l.ok" VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00004', 'Unable to encrypt transaction key: ' $ crypto_last_error() ) }">
    <MvELSE>
        <MvASSIGN NAME = "l.payment_secure:key"     VALUE = "{ crypto_base64_encode( l.key_encrypted ) }">
    </MvIF>
    <MvASSIGN NAME = "l.payment_key"        VALUE = "">
    <MvIF EXPR = "{ l.rsa }">
        <MvASSIGN NAME = "l.null"           VALUE = "{ rsa_free( l.rsa ) }">
    </MvIF>
    <MvFUNCTIONRETURN VALUE = "{ l.ok }">
</MvFUNCTION>
 
 
<MvFUNCTION NAME = "Decrypt_Payment" PARAMETERS = "crypt_id, key, secure_data var, data var, passphrase" STANDARDOUTPUTLEVEL = "">
    <MvIF EXPR = "{ l.crypt_id }">
        <MvIF EXPR = "{ len( l.secure_data ) }">
            <MvIF EXPR = "{ NOT len( l.passphrase ) }">
                <MvASSIGN NAME = "l.data" VALUE = "">
                <MvFUNCTIONRETURN VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00005', 'Pass Phrase Missing. Decryption Failed.' ) }">
            </MvIF>
            <MvIF EXPR = "{ NOT [ g.Library_Filename_DB ].Encryption_Load_ID( l.crypt_id, l.encryption ) }">
                <MvASSIGN NAME = "l.data" VALUE = "">
                <MvFUNCTIONRETURN VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00006', 'Error Loading Encryption Key. Decryption Failed.' ) }">
            </MvIF>
            <MvIF EXPR = "{ NOT rsa_load_privatekey_mem( l.encryption:prv_key, l.rsa, l.passphrase ) }">
                <MvASSIGN NAME = "l.data" VALUE = "">
                <MvFUNCTIONRETURN VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00007', 'Unable to load private key for decryption. Decryption Failed.' ) }">
            </MvIF>
            <MvASSIGN NAME = "l.key_encrypted" VALUE = "{ crypto_base64_decode( l.key ) }">
            <MvIF EXPR = "{ NOT rsa_private_decrypt( l.rsa, l.key_encrypted, l.key_decrypted ) }">
                <MvASSIGN NAME = "l.data" VALUE = "">
                <MvFUNCTIONRETURN VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00008', 'Order key decryption failed. Decryption Failed.' ) }">
            </MvIF>
            <MvASSIGN NAME = "l.data_decoded" VALUE = "{ crypto_base64_decode( l.secure_data ) }">
            <MvIF EXPR = "{ NOT bf_decrypt( l.key_decrypted, l.data_decoded, l.data_decrypted ) }">
                <MvASSIGN NAME = "l.data" VALUE = "">
                <MvFUNCTIONRETURN VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00009', 'Order data decryption failed. Decryption Failed.' ) }">
            </MvIF>
            <MvASSIGN NAME = "l.data" VALUE = "{ miva_array_deserialize( l.data_decrypted ) }">
            <MvIF EXPR = "{ l.rsa }">
                <MvIF EXPR = "{ NOT rsa_free( l.rsa ) }">
                    <MvASSIGN NAME = "l.data" VALUE = "">
                    <MvFUNCTIONRETURN VALUE = "{ [ g.Module_Library_Utilities ].Error( 'MER-CRY-00010', 'RSA free key failed' ) }">
                </MvIF>
            </MvIF>
        </MvIF>
    <MvELSE>
        <MvASSIGN NAME = "l.data" VALUE = "{ miva_array_deserialize( l.secure_data ) }">
    </MvIF>
    <MvFUNCTIONRETURN VALUE = 1>
</MvFUNCTION>

[Kent Multer]

Re: List of MivaScript builtin functions and discussion thereof

Hi Burch --

I checked out the references to OpenSSL docs, and this isn't really a complete solution for learning how to do encryption and decryption in Miva Script. Those docs seem to assume that the reader already knows all the math, and just needs documentation on things like which parameter to use for the public exponent. For those of us who don't know what a public exponent is, or what its value should be, this isn't much help.

The notes you gave on Merchant's built-in encryption functions are excellent. If you want to complete the doc package with one click, you can just publish the source code for those two functions. That'll tell the developers everything we need to know about how to use the Miva encryption functions in a real-world way.

Thanks --

[burch]

Re: List of MivaScript builtin functions and discussion thereof

Any other info available on the xml_parse functions? I know about xml_parse() but would like to know more about the other xml_parse functions, i cannot find them in any release notes i have. I'm right in the middle of some XML parsing and these might come in very useful.

Thanks again!

There's some documentation in the 1.17 Release Notes, but basically those functions provide a way to partially parse very large XML files, saving the internal state of the XML parser so you can resume the parse on a subsequent page load. We added these mechanisms for the MM5 provisioning system so that it could deal with very large provisioning files.

Here's the old docs (including code samples):

int xml_parse_section_init( filepath, location, level )
 Initializes a xml_parse_section session.
 filepath - source file path (same as xml_parse)
 location - 'script' or 'data' (same as xml_parse)
 level  - level (not including the root) to return a "section"
      of parsed XML output.
 returns 1 on success, 0 on failure.
--------------------------------------------------------------------------------
int xml_parse_section( output var, eof var)
 return a section of XML
 output - Parsed XML output, same format at xml_parse().
 eof  - boolean "end of file"
 returns 1 on success, 0 on failure.
 The output will be as if the subtree at the point that 
 parsing paused was the only content of the file.
 Example:
 <root>
  <one>
   <two>
    <three>data</three>
   </two>
  </one>
  <four>
   <five>
    <six>more data</six>
   </five>
  </four>
 </root>
 with a xml_parse_section_init level of two will result in xml_parse_section
 running twice with output, with the third time returning 1 in l.eof.
 The resulting xml output will look as if these two separate files were
 parsed:
 <root>
  <one>
   <two>
    <three>data</three>
   </two>
  </one>
 </root>
 ... and ...
 <root>
  <four>
   <five>
    <six>more data</six>
   </five>
  </four>
 </root>
--------------------------------------------------------------------------------
void xml_parse_error( lineno var, errortext var )
 Retrieves error information for xml_parse, xml_parse_section, and 
 xml_parse_section_init
 lineno - line number of the XML file the error occurred (if applicable)
 errortext - text of the error.
 returns nothing.
 lineno will be zero if the error was at file open time (xml_parse and 
 xml_parse_section_init).
--------------------------------------------------------------------------------
int xml_parse_section_getstate( target var )
 Retrieves parse state information from a xml_parse_section session.
 target - output aggregate.
 returns 1 on success, 0 on failure.
 The target variable will contain information useful for restarting an
 xml_parse_section session.  This information is suitable for MvHIDEing
 and passing to a new iteration of the VM.
--------------------------------------------------------------------------------
 xml_parse_section_setstate( source var )
 Sets internal parse state information for a xml_parse_section session.
 source - input aggregate.
 returns 1 on success, 0 on failure.
 The source variable is the same (or reconstituted via MvHIDE or
 serialize/deserialize) variable retutned from xml_parse_section_getstate.
 
--------------------------------------------------------------------------------
Here is an example showing these new functions working together:
<MvIF EXPR = "{ g.Loaded }">
    xml_parse_section_setstate = <MvEVAL EXPR = "{ xml_parse_section_setstate( g.State ) }"><br> <MvELSE>
    xml_parse_section_init = <MvEVAL EXPR = "{ xml_parse_section_init( 'provide.xml', 'data', 2 ) }"><br> </MvIF>
<MvIF EXPR = "{ NOT xml_parse_section( l.output, l.eof ) }">
    DEBUG: xml_parse_section failure<br>
    <MvEVAL EXPR = "{ xml_parse_error( l.line, l.text) }">
    Error line = <MvEVAL EXPR = "{ l.line }"><br>
    Error fetch = <MvEVAL EXPR = "{ l.text }"><br>
    <MvEXIT>
</MvIF>
<MvEVAL EXPR = "{ l.output:PROVISION:CHILDREN[1]:NAME }"> <MvEVAL EXPR = "{ l.output:PROVISION:CHILDREN[1]:ATTRIBUTE:CODE }"> <MvEVAL EXPR = "{ l.output:PROVISION:CHILDREN[1]:CHILDREN[1]:LINENO }"> <MvEVAL EXPR = "{ l.output:PROVISION:CHILDREN[1]:CHILDREN[1]:NAME
}"><br>
<MvIF EXPR = "{ NOT l.eof }">
    xml_parse_section_getstate = <MvEVAL EXPR = "{ xml_parse_section_getstate( g.State ) }"><br>
    <MvASSIGN NAME = "g.Loaded" VALUE = 1>
    <form method="post">
    <MvHIDE FIELDS = "g.Loaded, g.State">
    <input type="submit" value="Next">
    </form>
    <hr>
    <MvEVAL EXPR = "{ glosub( miva_array_serialize( l.output ), ',', '<br>' ) }">
    <hr>
<MvELSE>
    eof found<br>
</MvIF>

[burch]

Re: List of MivaScript builtin functions and discussion thereof

Hi Jon -- thanks again for your work on this.

In my experience, the biggest gap in the documentation is with the encryption & decryption functions -- the ones in Merchant, as well as the Miva built-ins. The documentation is so sketchy, it's really impossible to use these functions without a lot of trial and error. And encryption is something that more and more of us will be working with, as the world gets more security-conscious.

Most of the crypto functions are direct analogs of the underlying OpenSSL library functions. Is there a particular one you're interested in?

We use the rsa_ and bf_ (blowfish) functions for order encryption in Miva Merchant, and crypto_hmac_sha1 is used by a lot of the third party payment solutions (Google Checkout, Checkout by Amazon, etc...) to verify requests that are passed through a shopper's browser.

OpenSSL's documentation on the Blowfish functions is here: http://www.openssl.org/docs/crypto/blowfish.html#

OpenSSL's documentation on the RSA functions is here: http://www.openssl.org/docs/crypto/rsa.html#

If you have a specific task you're trying to accomplish I'd be happy to put together some code examples.

lib/crypto.mv in Miva Merchant offers two wrapper functions that implement the specifics of the MM order encryption. The function declarations are:

<MvFUNCTION NAME = "Encrypt_Payment" PARAMETERS = "crypt_id, secure_data var, payment_secure var" STANDARDOUTPUTLEVEL = "">
<MvFUNCTION NAME = "Decrypt_Payment" PARAMETERS = "crypt_id, key, secure_data var, data var, passphrase" STANDARDOUTPUTLEVEL = "">

crypt_id is an id from the sNN_Encryption table. For encrypt, you would usually pass g.Store:crypt_id, which is the id of the current encryption key. On decryption, you would pass pay_secid from the Orders record you're working with.

key is the encrypted BlowFish key. It's only used for decryption. You get this key from Encrypt_Payment in the payment_secure output parameter.

secure_data is the data you want to encrypt. For Orders, we pass the serialized secure_data aggregate from the payment module.

payment_secure is an output variable that gets the (encrypted) randomly generated BlowFish key (l.payment_secure:key) and the encrypted data (l.payment_secure:data).

data is where the decrypted (but still serialized) data is stored by Decrypt_Payment.

passphrase is the passphrase for the encryption key identified by crypt_id. You must prompt the user for this passphrase, and it would be an extremely bad idea to ever store it in any form on the server. You can get the user's configured passphrase prompt from the sNN_Encryption table using the DB function Encryption_Load_ID( id, encryption var ).

Some background is probably required. Order data in Miva Merchant is encrypted using BlowFish. We randomly generate a BlowFish key for each order, then encrypt that key with the public key component of an RSA keypair. This lets us place orders without requiring any sort of stored password/key. To decrypt an order, we use the private key component of the same RSA keypair (which is stored on the server encrypted with the key passphrase) to decrypt the order-specific BlowFish key, which is then used the decrypt the order data.

[BillBuilt]

Re: List of MivaScript builtin functions and discussion thereof

Any other info available on the xml_parse functions? I know about xml_parse() but would like to know more about the other xml_parse functions, i cannot find them in any release notes i have. I'm right in the middle of some XML parsing and these might come in very useful.

Thanks again!

[Kent Multer]

Re: List of MivaScript builtin functions and discussion thereof

Hi Jon -- thanks again for your work on this.

In my experience, the biggest gap in the documentation is with the encryption & decryption functions -- the ones in Merchant, as well as the Miva built-ins. The documentation is so sketchy, it's really impossible to use these functions without a lot of trial and error. And encryption is something that more and more of us will be working with, as the world gets more security-conscious.

[burch]

Re: List of MivaScript builtin functions and discussion thereof

tar_directory() is like dir()--it gives you an array of structures that describe the contents of the file. Its output is like this:

[n]:name
[n]:mode
[n]:uid
[n]:gid
[n]:size
[n]:timestamp
[n]:type
[n]:link
[n]:user
[n]:group
[n]:majordev
[n]:minordev

Those fields are basically straight out of the tar header. And yes, if you wanted to include several directories, you'd have to build the directory structure in a temporary folder and then tar_create() that.

The 5.00 VM Release Notes contain this description of wdownload:

o  A new built-in function wdownload takes the first three parameters as wget, and acts the same as wget.
    It takes in addition a fourth and fifth parameter.  If the fourth
   parameter is not blank, it must be the name of a function defined in the same
   compiled file as the call to wdownload, which takes three parameters:

        functionname( size, total, data var )

            size    - The current number of bytes read.
            total   - The expected size of the completed file.
            data    - The variable passed to wget as it's fifth parameter.

        Returns
            0 to abort the wget, 1 otherwise.

    If the callback is called, the global timeout will be reset when it returns back to wget.

wget() works basically the same way but doesn't provide a callback function. Looking at the release notes, it appears we originally were going to add the callback to wget(), but instead decided to implement wdownload() as a separate function.

[BillBuilt]

Re: List of MivaScript builtin functions and discussion thereof

This is great! How about some info on the archive.so functions.
I've used xml_parse(), but i did not know about the other xml_parse_...() functions. I bet they would've (and will) come in quite handy.

I've never used tar_create mainly because i didn't know about tar_extract, so if i had to use another language to extract, i may as well use it to create too. However, with the introduction of tar_directory, i'm now confused as to what it is for, since tar_create is supposed to archive directories. Also, if i wanted to tar several files in different directories, would i need to first copy them to a temp folder and then tar that folder?

As far as wget/wdownload, i'd just like to know more about them and their parameters.

Thanks!!

[burch]

Re: List of MivaScript builtin functions and discussion thereof

The keyword functions were originally developed for a keyword-based search algorithm we had in development for some long-since-lost version of Miva Merchant.

They only implement a very basic keyword extraction algorithm that looks for text in an HTML document (including tag attributes) and then stems it. So sadly, they're not a replacement for regexp or even FMT.

There's a fairly detailed description of all the keyword_xxx functions in the v4.12 VM release notes. I'll include that section below:

 keyword_extract:
  Given a string, will extract keywords from it.  Skipping SGML tags and
  stemming words (e.g., "running" becomes "run", but "bring" is not
  changed), the unique list of keywords are placed in the keywords
  as an array.  Also, some common english words (the articles, "for"
  "are", etc.) are removed.
     Note that there are some words may stem unexpectedly (e.g.,
  "has" transforms to "ha").
  Syntax:
   keyword_extract( string , keywords )
 
  Parameters:
   string:  Source string
   keywords: array of keywords
  Return Value:
   Count of resulting keywords in the "keywords" array.
   
 keyword_in
  Performs a keyword_extract (see above) on the "string" parameter, 
  and determines if the value in the "keywords" parameter is
  contained in the keyword list, and returns a boolean "1" or 
  "0" to signify that the keyword is or is not in the string.
  If the "keywords" parameter is an array of strings, checks each
  value in the array, and returns an array of booleans specifying
  whether the array element is or is not in the string.
  
  Syntax:
   keyword_in( keywords , string )
 
  Parameters:
   string:  Source string
   keywords: String or array of keywords to check.
  Return Value:
   If keywords is an array, an array of booleans specifying if
   a given keyword element is in the keyword list.
   If keywords is a string, a boolean specifying if the
   given keyword is in the keyword list.
 keyword_extract_merge_init
  Initializes persistent storage for a multiple calls to
  keyword_merge_extract.  Note that this persistent storage
  is usable for multiple calls within a running Miva Script
  program, and is deleted by the VM at the end of program execution.
  
  
  Syntax:
   keyword_extract_merge_init( )
 
  Parameters:
   None.
  Return Value:
   None.

 keyword_extract_merge
  Extracts keywords as described in keyword_extract, but inserts
  them into a persistent array initialized by
  keyword_extract_merge_init.  The weight value passed in is
  associated with the results from the current string being analyzed.
  Syntax:
   keyword_extract_merge( string, weight )
 
  Parameters:
   string:  String to analyze for keywords.
   weight:  Weight to associate with the keywords found in 
      string in the persistent array.
  Return Value:
   None

 keyword_extract_merge_results
  Returns results from one or more keyword_extract_merge calls,
  storing them as an aggregate in "keywords".
  
  Syntax:
   keyword_extract_merge_results( keywords )
 
  Parameters:
   keywords: An aggregate with the following format, in 
      keyword order:
      keywords[ n ]:stemmed
       The stemmed version of the keyword.  All subkeys
       may be different text, but stemmed to this value.
      
      keywords[ n ]:weight
       Cumulative weight of this keyword.  If a 
       weight of 1 is used for all calls to
       keyword_extract_merge, this will be the total count 
       of references to this keyword.
      keywords[ n ]:subkey_count
       Count of other references to this key.
      keywords[ n ]:subkey[ n ]:keyword
       Keyword (unstemmed, the stemmed value is in 
       keywords[ n ]:stemmed).
      keywords[ n ]:subkey[ n ]:weight
       Weight of this keyword reference.  
  Return Value:
   Number of keywords.

[Emma]

Re: List of MivaScript builtin functions and discussion thereof

I don't Spam I promise :) I'm searching for a way to get to the javascript (and many other languages) "regexp" function. There was FMT but it's gone and I would like to understand if the keyword functions can do such jobs.

PS: I developed a fake regexp function but I'm forced to use MvWhile and I would like to find another way to do that (when I have lot of text to parse, it's not very quick).

Thanks again
Claudiu