[Setup] cut or copy and paste "large" plone site?

[John Schinnerer]

Hi,

I think this is the place for install/upgrade issues. If not please
direct me to the appropriate place.

Plone 3.3.5 is the platform here, but the reason for the question is to
facilitate upgrading a 3.3.5 site to 4.x.

Is it possible to cut/copy and paste a "large" plone site, within a
given zope instance that has one or more existing plone sites?

Also, is there a way to verify that "clear and rebuild" and "update" of
catalog (portal_catalog -> Advanced tab) complete, or not, for a "large"
site?

"Large" in this case means a Data.fs of ~5.7GB.

I am asking because I have started "clear and rebuild" and "update"
operations multiple times, but after some long time of the browser
"waiting for domain.tld...," the right pane of the ZMI refreshes with a
"connection refused."

I am getting the same result when I try and copy or cut and paste the
site. In that case it seems clear the operation failed at some point,
since the site remains in the original location when I refresh the ZMI
tab (was not moved or copied).

Does one of the logs track these operations? Or is there any other way
to find out what is actually happening, especially with the catalog
operations?

thanks,
John S.

[Steve McMahon]

You should find an event.log file in your client directories. Sometimes put
in a separate log directory.

You may wish to start a client in foreground mode and directly address it
while doing this kind of experimenting.

Watching the log or foreground console will give you a pretty good idea of
whether or not the clear/rebuild is reindexing. If your site is not just
large in bulk, but has a lot of content objects, you'll need to enable
subtransactions.

Other strategies:

* Set up a mirror of the old site (using backup data) and simply delete
everything but the target site. If you want to combine it others at the end
of the update process, you may use mount points.

* Use Zope's export/import facility, restoring to a mirror of the old setup
(same versions/add-ons). If you restore to a different folder, be ready to
rebuild the catalog.

Hi,
I think this is the place for install/upgrade issues. If not please direct
me to the appropriate place.
Plone 3.3.5 is the platform here, but the reason for the question is to
facilitate upgrading a 3.3.5 site to 4.x.
Is it possible to cut/copy and paste a "large" plone site, within a given
zope instance that has one or more existing plone sites?
Also, is there a way to verify that "clear and rebuild" and "update" of
catalog (portal_catalog -> Advanced tab) complete, or not, for a "large"
site?
"Large" in this case means a Data.fs of ~5.7GB.
I am asking because I have started "clear and rebuild" and "update"
operations multiple times, but after some long time of the browser "waiting
for domain.tld...," the right pane of the ZMI refreshes with a "connection
refused."
I am getting the same result when I try and copy or cut and paste the
site. In that case it seems clear the operation failed at some point, since
the site remains in the original location when I refresh the ZMI tab (was
not moved or copied).
Does one of the logs track these operations? Or is there any other way to
find out what is actually happening, especially with the catalog operations?
thanks,
John S.
--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
https://lists.plone.org/mailman/listinfo/plone-setup

[John Schinnerer]

Thanks Steve, very helpful.

Post by Steve McMahon
You should find an event.log file in your client directories. Sometimes
put in a separate log directory.

Yep, found them. I tried both cut/paste and copy/paste of the site.

Log shows that the cut/paste attempt apparently ended after a whole lot
of these errors (too many open files):
--------
2015-12-15T08:31:35 INFO Archetypes Error while trying to convert file
contents to 'text/plain' in <Field file(file:rw)>.getIndexable() of
<ATFile at
/copy_of_hawaiicountycdp.info/hilo/bayfront-trails/Public_Meeting_2
_Presentation-Press_Quality.pdf>: [Errno 24] Too many open files:
'/tmp/tmpktmYI0'
--------

The copy/paste ended with this error, in the ZMI and detailed in the log:
----
Error Type: ImportError
Error Value: No module named collective.recaptcha.settings
--------------

...which I have also gotten trying to import the same site into a
different location in the same instance, after exporting it successfully.
So it looks like copy/paste does an export/import behind the scenes,
while cut/paste does...something else...?

Re the ImportError - the 3.3.5 site is an upgrade from a 3.2.2 site that
used collective.recaptcha. In that upgrade process, I found that adding
collective.recaptcha to the 3.3.5 buildout caused an error on every page
render attempt. It was used on only two forms, not mission critical at
all, so I took it back out of the 3.3.5 buildout. The two forms then
threw errors rendering, but after I deleted the recaptcha objects from
the form folders they worked again.
I thought that was the end of it, but apparently there is some kind of
collective.recaptcha ghost in the machine, as it is still trying to
import a module.
No idea what to do about that, suggestions welcome.

Post by Steve McMahon
Watching the log or foreground console will give you a pretty good idea
of whether or not the clear/rebuild is reindexing. If your site is not
just large in bulk, but has a lot of content objects, you'll need to
enable subtransactions.

Subtransactions are enabled.
The threshold is set at 10000 (not by me - some kind of default?) - the
catalog has 9039 records - so I assume that means it will do the whole
catalog in one transaction, unless I lower the threshold?

I will post separately w/new subject line for the rest of this since it
leads to what seems to be the core issue (cut/copy & paste the site
being an attempt to work around that issue that is not working).

thanks,
John S.

Post by Steve McMahon
Hi,
I think this is the place for install/upgrade issues. If not please
direct me to the appropriate place.
Plone 3.3.5 is the platform here, but the reason for the question is
to facilitate upgrading a 3.3.5 site to 4.x.
Is it possible to cut/copy and paste a "large" plone site, within a
given zope instance that has one or more existing plone sites?
Also, is there a way to verify that "clear and rebuild" and "update"
of catalog (portal_catalog -> Advanced tab) complete, or not, for a
"large" site?
"Large" in this case means a Data.fs of ~5.7GB.
I am asking because I have started "clear and rebuild" and "update"
operations multiple times, but after some long time of the browser
"waiting for domain.tld...," the right pane of the ZMI refreshes
with a "connection refused."
I am getting the same result when I try and copy or cut and paste
the site. In that case it seems clear the operation failed at some
point, since the site remains in the original location when I
refresh the ZMI tab (was not moved or copied).
Does one of the logs track these operations? Or is there any other
way to find out what is actually happening, especially with the
catalog operations?
thanks,
John S.
--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
<tel:[phone]>
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
https://lists.plone.org/mailman/listinfo/plone-setup
_______________________________________________
Setup mailing list
https://lists.plone.org/mailman/listinfo/plone-setup

[John Schinnerer]

Hi again,

Here is what I was hoping to work around, possibly with cut/copy and
paste of the site.

Post by Steve McMahon
* Set up a mirror of the old site (using backup data) and simply delete
everything but the target site. If you want to combine it others at the
end of the update process, you may use mount points.

Not the problem (multiple sites in one Data.fs). It's just one site in
one Data.fs I'm dealing with here.

Post by Steve McMahon
* Use Zope's export/import facility, restoring to a mirror of the old
setup (same versions/add-ons). If you restore to a different folder, be
ready to rebuild the catalog.

Tried this also. The export works OK; import hits the ImportError I
mentioned in previous post - looking for something in
collective.recaptcha, removed in upgrade because in the 3.3.5 it causes
an error (with a huge mass of code output, not a typical orderly
traceback) on every page render attempt.

The bottom line upgrade problem, which I've failed to work around with
export/import or cut/copy-paste, is that when I put the 3.3.5 Data.fs
into the 4.0.8 instance and dry run the upgrade, I get a keyError on a
traversal. Specifically:

[lots of upgrade steps completing OK]
...
Ran upgrade step: Upgrade DCMI metadata storage in portal_metadata
Ran upgrade step: Add the recursive groups PAS plugin.
Upgrade aborted. Error:
Traceback (most recent call last):
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Plone-4.0.8-py2.6.egg/Products/CMFPlone/MigrationTool.py",
line 175, in upgrade
step['step'].doStep(setup)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Products.GenericSetup-1.6.3-py2.6.egg/Products/GenericSetup/upgrade.py",
line 142, in doStep
self.handler(tool)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
line 480, in updateLargeFolderType
update(brain)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
line 473, in update
obj = brain.getObject()
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/Products/ZCatalog/CatalogBrains.py",
line 83, in getObject
parent = parent.unrestrictedTraverse(path[:-1])
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/OFS/Traversable.py",
line 293, in unrestrictedTraverse
raise e
KeyError: 'puna-cdp'

The most likely explanation I have found so far is in this thread:
http://sourceforge.net/p/plone/mailman/message/27276817/

Specifically the part about paths to objects not matching:
"The error you are getting is...happening while the migration script is
trying to find a particular piece of content...by moving through the
path elements leading to that object. 'KeyError'...is raised when
dictionary lookup on an object fails to find an expected key in the
dictionary...you used to have the site on a separate Data.fs...on a
mountpoint...my guess is that the ZODB has persisted path information
for that site and everything in it that includes the mountpoint...it may
be that the ZODB and its catalog think the site is still in the old
location and are looking for it there. Since it is not there, you get a
KeyError."

This Data.fs I am trying to upgrade has one migration/upgrade in its
history that I did not do. It was a server OS upgrade during which my
sysadmin did something while reinstalling/re-configuring that led to a
path issue.

Before, there was a ZODB mount point 'domain.tld' in the ZMI root folder
(configured in buildout as usual) and in that 'domain.tld' folder was
the plone site, like:
/domain.tld/plone-site

After, that somehow changed to have one more 'domain.tld' folder in the
path in the ZMI, like:
/domain.tld/domain.tld/plone-site

BUT on the file system (as configured in buildout all along) the mount
point was still just
domain.tld/Data.fs

In other words, when I copy this Data.fs into
domain.tld/Data.fs
on the file system, the result is
domain.tld/domain.tld/plone-site
in the ZMI.

I've tried to fix by export/import and recatalog, but import hits the
collective.recaptcha import error (with collective.recaptcha breaking
the 3.3.5 site if I add it). I tried leaving collective.recaptcha in and
dry-running upgrade, just to see what would happen. That fixed the
collective.recaptcha import error but then gave an ImportError on
registry.interfaces. No idea why or what to do with that.

Then I tried to fix by cut/copy paste to move the site and then
recatalog, with the copy/paste failing on import for same reasons as
above, and the cut/paste failing for reasons yet TBD.

I also tried to create the duplicate-directory mount point path in the
4.0.8 instance, without success.

If I have this in buildout:
---------------
[filestorage]
recipe = collective.recipe.filestorage
blob-storage = var/blobstorage-%(fs_part_name)s
parts =
domain.tld
----------------

...it works just like in the 3.3.5 instance, namely, the filesystem
mount point path is
domain.tld/Data.fs

I tried changing it to

---------------
...
parts =
domain.tld/domain.tld
------------------

...but that gave me a triplicate path on the file system!
In other words, when I went to add the mount point in the ZMI, it showed as
domain.tld/domain.tld/domain.tld

No idea why or if that's a bug (seems like one, but maybe it's not even
allowed to have a multi-part path for the 'parts' declaration, or the /
has to be escaped, or...?).
In any case, it didn't get me a duplicate of the internal path, so I
don't know if that would work if it did.

So...what to try next?
I am about out of ideas.
Or, I'm thinking of something rather over my head, like a script that
would go through the whole catalog and fix the path strangeness, remove
the duplicate path element, something like that (if that's even possible).

Any assistance appreciated.

thanks,
John S.

Post by Steve McMahon
Hi,
I think this is the place for install/upgrade issues. If not please
direct me to the appropriate place.
Plone 3.3.5 is the platform here, but the reason for the question is
to facilitate upgrading a 3.3.5 site to 4.x.
Is it possible to cut/copy and paste a "large" plone site, within a
given zope instance that has one or more existing plone sites?
Also, is there a way to verify that "clear and rebuild" and "update"
of catalog (portal_catalog -> Advanced tab) complete, or not, for a
"large" site?
"Large" in this case means a Data.fs of ~5.7GB.
I am asking because I have started "clear and rebuild" and "update"
operations multiple times, but after some long time of the browser
"waiting for domain.tld...," the right pane of the ZMI refreshes
with a "connection refused."
I am getting the same result when I try and copy or cut and paste
the site. In that case it seems clear the operation failed at some
point, since the site remains in the original location when I
refresh the ZMI tab (was not moved or copied).
Does one of the logs track these operations? Or is there any other
way to find out what is actually happening, especially with the
catalog operations?
thanks,
John S.
--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
<tel:[phone]>
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
https://lists.plone.org/mailman/listinfo/plone-setup
_______________________________________________
Setup mailing list
https://lists.plone.org/mailman/listinfo/plone-setup

[John Schinnerer]

I've managed to get past this - posting my solution in case it helps
someone else somewhere somehow sometime.

By setting the sub-transactions threshold in the portal_catalog Advanced
tab to a relatively small number, I was able to get catalog update, and
catalog clear/rebuild, to finish successfully.

Default appeared to be 10000; I tried 1000 but it still failed with the
"too many files open" error in the log.
I just took it down to 100 for a next try and that worked. This was on a
test/dev install of the site so it didn't matter much if that setting
would have a negative effect on the production site. It is only for
getting through the upgrade tangles and challenges.

Once the catalog update and clear/rebuild were done, I copied the
Data.fs from the 3.3.5 site to the 4.0.8 site and the upgrade completed
successfully. No more keyError.

cheers,
John S.

Post by John Schinnerer
Hi again,
Here is what I was hoping to work around, possibly with cut/copy and
paste of the site.

Post by Steve McMahon
* Set up a mirror of the old site (using backup data) and simply delete
everything but the target site. If you want to combine it others at the
end of the update process, you may use mount points.

Not the problem (multiple sites in one Data.fs). It's just one site in
one Data.fs I'm dealing with here.

Post by Steve McMahon
* Use Zope's export/import facility, restoring to a mirror of the old
setup (same versions/add-ons). If you restore to a different folder, be
ready to rebuild the catalog.

Tried this also. The export works OK; import hits the ImportError I
mentioned in previous post - looking for something in
collective.recaptcha, removed in upgrade because in the 3.3.5 it causes
an error (with a huge mass of code output, not a typical orderly
traceback) on every page render attempt.
The bottom line upgrade problem, which I've failed to work around with
export/import or cut/copy-paste, is that when I put the 3.3.5 Data.fs
into the 4.0.8 instance and dry run the upgrade, I get a keyError on a
[lots of upgrade steps completing OK]
...
Ran upgrade step: Upgrade DCMI metadata storage in portal_metadata
Ran upgrade step: Add the recursive groups PAS plugin.
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Plone-4.0.8-py2.6.egg/Products/CMFPlone/MigrationTool.py",
line 175, in upgrade
step['step'].doStep(setup)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Products.GenericSetup-1.6.3-py2.6.egg/Products/GenericSetup/upgrade.py",
line 142, in doStep
self.handler(tool)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
line 480, in updateLargeFolderType
update(brain)
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
line 473, in update
obj = brain.getObject()
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/Products/ZCatalog/CatalogBrains.py",
line 83, in getObject
parent = parent.unrestrictedTraverse(path[:-1])
File
"/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/OFS/Traversable.py",
line 293, in unrestrictedTraverse
raise e
KeyError: 'puna-cdp'
http://sourceforge.net/p/plone/mailman/message/27276817/
"The error you are getting is...happening while the migration script is
trying to find a particular piece of content...by moving through the
path elements leading to that object. 'KeyError'...is raised when
dictionary lookup on an object fails to find an expected key in the
dictionary...you used to have the site on a separate Data.fs...on a
mountpoint...my guess is that the ZODB has persisted path information
for that site and everything in it that includes the mountpoint...it may
be that the ZODB and its catalog think the site is still in the old
location and are looking for it there. Since it is not there, you get a
KeyError."
This Data.fs I am trying to upgrade has one migration/upgrade in its
history that I did not do. It was a server OS upgrade during which my
sysadmin did something while reinstalling/re-configuring that led to a
path issue.
Before, there was a ZODB mount point 'domain.tld' in the ZMI root folder
(configured in buildout as usual) and in that 'domain.tld' folder was
/domain.tld/plone-site
After, that somehow changed to have one more 'domain.tld' folder in the
/domain.tld/domain.tld/plone-site
BUT on the file system (as configured in buildout all along) the mount
point was still just
domain.tld/Data.fs
In other words, when I copy this Data.fs into
domain.tld/Data.fs
on the file system, the result is
domain.tld/domain.tld/plone-site
in the ZMI.
I've tried to fix by export/import and recatalog, but import hits the
collective.recaptcha import error (with collective.recaptcha breaking
the 3.3.5 site if I add it). I tried leaving collective.recaptcha in and
dry-running upgrade, just to see what would happen. That fixed the
collective.recaptcha import error but then gave an ImportError on
registry.interfaces. No idea why or what to do with that.
Then I tried to fix by cut/copy paste to move the site and then
recatalog, with the copy/paste failing on import for same reasons as
above, and the cut/paste failing for reasons yet TBD.
I also tried to create the duplicate-directory mount point path in the
4.0.8 instance, without success.
---------------
[filestorage]
recipe = collective.recipe.filestorage
blob-storage = var/blobstorage-%(fs_part_name)s
parts =
domain.tld
----------------
...it works just like in the 3.3.5 instance, namely, the filesystem
mount point path is
domain.tld/Data.fs
I tried changing it to
---------------
...
parts =
domain.tld/domain.tld
------------------
...but that gave me a triplicate path on the file system!
In other words, when I went to add the mount point in the ZMI, it showed as
domain.tld/domain.tld/domain.tld
No idea why or if that's a bug (seems like one, but maybe it's not even
allowed to have a multi-part path for the 'parts' declaration, or the /
has to be escaped, or...?).
In any case, it didn't get me a duplicate of the internal path, so I
don't know if that would work if it did.
So...what to try next?
I am about out of ideas.
Or, I'm thinking of something rather over my head, like a script that
would go through the whole catalog and fix the path strangeness, remove
the duplicate path element, something like that (if that's even possible).
Any assistance appreciated.
thanks,
John S.

Post by Steve McMahon
Hi,
I think this is the place for install/upgrade issues. If not please
direct me to the appropriate place.
Plone 3.3.5 is the platform here, but the reason for the question is
to facilitate upgrading a 3.3.5 site to 4.x.
Is it possible to cut/copy and paste a "large" plone site, within a
given zope instance that has one or more existing plone sites?
Also, is there a way to verify that "clear and rebuild" and "update"
of catalog (portal_catalog -> Advanced tab) complete, or not, for a
"large" site?
"Large" in this case means a Data.fs of ~5.7GB.
I am asking because I have started "clear and rebuild" and "update"
operations multiple times, but after some long time of the browser
"waiting for domain.tld...," the right pane of the ZMI refreshes
with a "connection refused."
I am getting the same result when I try and copy or cut and paste
the site. In that case it seems clear the operation failed at some
point, since the site remains in the original location when I
refresh the ZMI tab (was not moved or copied).
Does one of the logs track these operations? Or is there any other
way to find out what is actually happening, especially with the
catalog operations?
thanks,
John S.
--
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
<tel:[phone]>
http://eco-living.net
http://sociocracyconsulting.com
_______________________________________________
Setup mailing list
https://lists.plone.org/mailman/listinfo/plone-setup
_______________________________________________
Setup mailing list
https://lists.plone.org/mailman/listinfo/plone-setup