Conferences & Events
Events for all Levels and InterestsStay
Jump Start Your Career GrowthStay
Get on the Higher Ed IT MapStay
Uncommon Thinking for the Common Good™Stay
warning about GAMME, "malformed" account provisioning, and lost email
I just wanted to share with you the need to be extra careful with initial account provisioning and GAMME migration. We just had a situation where 3 student accounts out of 1000 provisioned recently were "malformed" at provisioning time due to a bug at Google. The provisioning was done via the Provisioning API with no errors. A GAMME migration was run and performed flawlessly, showing no errors (our partner parses all GAMME logs and provides us with an exception report of any messages that failed to migrate). New Email for the students was accepted for a week, showing no errors. However, when the students tried to login, they got a 500 error and were unable to get to their accounts. Further, when Google "fixed" their accounts, none of their migrated mail was there, nor was any of their received email post-migration. It was apparently queued internally at Google and may have started bouncing back after the typical 3 days or so of queue retries. It was initially accept by Gmail though and queued internally within Google, so our outbound SMTP logs do not see any errors either. Apparently this is now (as of last night) a known bug with GAMME as the nature of the beast is that the API used by GAMME did accept the messages and put them somewhere even though the account was not accesible via the web or IMAP and when fixed, that pointer to somewhere seems to have gone astray.
So, we are now performing a GAMME workaround (with our Google Apps partner) of checking the users' accounts after provisioning to confirm that they are in fact properly provisioned. Since we had just migrated, we still had the old mail on our Cyrus servers and remigrated it. BUT, we have lost a week's worth of new mail which Google is still trying to track down internally (as it was accepted and queued somewhere). We are also checking our outbound SMTP logs for the past 2 weeks and will be letting the students know who sent them messages (but not what was in them) so they have the option of contacting those individuals and asking them to resend. We also retrieved and resent the messages sent to them by our Sakai system so at least they get their official course-related emails. Once we identify any mailing list mail, we can also direct them to the archives for those mailing lists (where they exist).
Just a heads up to be extra careful.
I'm tired of hearing statistics about how this is only .3% of the user accounts and basically 0% of all Google accounts in the universe.
********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.