ProblemsWithHereDocuments

This topic shows problems that only appear on trunk, and will not render correctly on 1.0.x

Where Here-documents do not work

-- MichaelTempest - 04 Feb 2010

commonTagsHandler

Any plugin that uses the commonTagsHandler is vulnerable to problems. To check:
Non-core:
  • ActionTrackerPlugin
  • Lots more (WIP)

Vulnerable:

Other handlers

Other handlers may also be vulnerable to problems: To check:
Non-core:
  • EasyMacroPlugin: beforeCommonTagsHandler
  • SmartWordBreakPlugin: postRenderingHandler
  • Lots more (WIP)

Vulnerable:

FIXED: Parameters to preferences

It is not normal practice to pass parameters to %PREFERENCES%, but Here-documents should work the same as regular parameters.

Expected: The macro is working

The macro is working not HERE

EditTablePlugin

Click the table-edit button to see the problem

Title

| | This is a HERE document HERE | | So is this HERE

FIXED: STARTSECTION and ENDSECTION

There are three sections: pickle, egg and ham. The three sections are all within a <!--comment-->.
  • There is nothing unusual about pickle.
  • The ENDSECTION parameter for egg uses a HERE document.
  • The STARTSECTION parameter for ham uses a HERE document.

I eat Pickles, Egg egg HERE

Ham ham HERE

-->

I eat Pickles,
Warning
Can't INCLUDE Development.ProblemsWithHereDocuments repeatedly, topic is already included.
s and Warning: Can't find named section ham in topic Development.ProblemsWithHereDocuments .

WYSIWYG editing

WysiwygPlugin does not protect HERE-document markers.

Foswiki::Func::extractNameValuePair

This function returns a string. I do not see a clean way to extend this API to handle here-documents. This API is used by these extensions:


STARTSECTION and ENDSECTION are core macros, and should support heredocs, as should any macros that register a tag handler. The problem is macros that are handled in the commonTagsHandler, and IMHo you can't make an omelette without breaking eggs. If we don't ever change anything in case something breaks, we never change anything (sound familiar? we had enough of it in (tm) days)

-- CrawfordCurrie - 04 Feb 2010

Some thoughts:
  • Plugins that use a commonTagsHandler should never get to see the content of a HERE document similar to verbatim. they will only get to see it when the HERE document has been added to the parsing flow again when it is time to participate in completing the macro that started the HERE document
  • There are a couple of macros, that is pairs of macros that always appear together surrounding a block. Eg. RED...ENDCOLOR, STARTINCLUDE...STOPINCLUDE, STARTSECTION...ENDSECTION, TWISTY...ENDTWISTY. Question is: when there's a HERE document it might get in conflict with the linear order of the block of code it surrounds. Actually, this may get so weird that HERE documents on block macros probably don't make real sense in the first place and users will most probably not make use of these two features together. Still, the TML parser has to cope with this case in some way. Imho, converting a HERE document to a proper macro parameter should be as stupid as possible. My intuition says:
    • %RED{<<HERE} red %ENDCOLOR% not red HERE
      is equivalent to
       %RED{"red %ENDCOLOR% not here"}% 
      ... everything up to the next HERE becomes the _DEFAULT parameter. (html error, everything gets red) *
      Ham ham HERE
      is equivalent to
       ham"}% 
      ... no proper definition of Ham

-- MichaelDaum - 12 Feb 2010

Michael - it occurred to me rather late that your examples do not follow here document syntax::
The general syntax is << followed by a delimiting identifier, followed, starting on the next line, by the text to be quoted, and then closed by the same identifier on its own line.

I agree that using HERE documents together with block macros may be confusing, and I agree that users will probably not do that. However, I think the syntax should be internally consistent within Foswiki, regardless of where it is used, and it should be consistent with other languages' use of HERE documents.

I am not sure what you mean about commonTagsHandlers.

-- MichaelTempest - 13 Feb 2010

There are two core calls to dispatch commonTagsHandler. These immediately follow calls to innerExpandMacros which flatten out HERE documents. So a commonTagsHandler should never see unexpanded HERE document syntax.

However that is not the case for all the handlers. The before handlers are all at particular risk of being exposed to HERE documents. Some of these handlers process topic text, for example to detect tables anchored around macros e.g. %TABLE.

The main issue with HERE-document syntax is that it's impossible to guarantee not breaking "old" code, where (unavoidable) assumptions have been made about the rendering order. The same risk applies to any new usage of SECTION blocks, or any other block structuring technique; we are creating blocks where none existed before. There is no magic bullet.

-- CrawfordCurrie - 14 Feb 2010

My problem with *SECTION is that it cannot be queried. Being able to query them would require some more ContentAccessSyntax (wrt to writing queries) anyway. But HERE documents are even more limiting: they can't even be extracted, even if I knew there was one in a particular topic. But headings aren't even on the radar, where there's no syntax at all for working with them, even though this what 99% of our wiki users understand "sections" to be.

I'm probably the only foswiki developer trying to query/mashup topic data that has been structured by ordinary users, without a wiki app developer.

So I'm not proposing any change to this fine proposal.

But it would be nice to still consider extending functionality of SECTIONs as was discussed on the road to working on HERE documents, because I do believe people will want to pass mashable/query-able sections into macros eventually.

-- PaulHarvey - 14 Feb 2010s and Warning: Can't find named section ham in topic Development.ProblemsWithHereDocuments .

WYSIWYG editing

WysiwygPlugin does not protect HERE-document markers.

Foswiki::Func::extractNameValuePair

This function returns a string. I do not see a clean way to extend this API to handle here-documents. This API is used by these extensions:


STARTSECTION and ENDSECTION are core macros, and should support heredocs, as should any macros that register a tag handler. The problem is macros that are handled in the commonTagsHandler, and IMHo you can't make an omelette without breaking eggs. If we don't ever change anything in case something breaks, we never change anything (sound familiar? we had enough of it in (tm) days)

-- CrawfordCurrie - 04 Feb 2010

Some thoughts:
  • Plugins that use a commonTagsHandler should never get to see the content of a HERE document similar to verbatim. they will only get to see it when the HERE document has been added to the parsing flow again when it is time to participate in completing the macro that started the HERE document
  • There are a couple of macros, that is pairs of macros that always appear together surrounding a block. Eg. RED...ENDCOLOR, STARTINCLUDE...STOPINCLUDE, STARTSECTION...ENDSECTION, TWISTY...ENDTWISTY. Question is: when there's a HERE document it might get in conflict with the linear order of the block of code it surrounds. Actually, this may get so weird that HERE documents on block macros probably don't make real sense in the first place and users will most probably not make use of these two features together. Still, the TML parser has to cope with this case in some way. Imho, converting a HERE document to a proper macro parameter should be as stupid as possible. My intuition says:
    • %RED{<<HERE} red %ENDCOLOR% not red HERE
      is equivalent to
       %RED{"red %ENDCOLOR% not here"}% 
      ... everything up to the next HERE becomes the _DEFAULT parameter. (html error, everything gets red) *
      %STARTSECTION{<<HERE}%Ham%ENDSECTION{"ham"}% ham HERE
      is equivalent to
      %STARTSECTION{"Ham%ENDSECTION{"ham"}% ham"}% 
      ... no proper definition of Ham

-- MichaelDaum - 12 Feb 2010

Michael - it occurred to me rather late that your examples do not follow here document syntax::
The general syntax is << followed by a delimiting identifier, followed, starting on the next line, by the text to be quoted, and then closed by the same identifier on its own line.

I agree that using HERE documents together with block macros may be confusing, and I agree that users will probably not do that. However, I think the syntax should be internally consistent within Foswiki, regardless of where it is used, and it should be consistent with other languages' use of HERE documents.

I am not sure what you mean about commonTagsHandlers.

-- MichaelTempest - 13 Feb 2010

There are two core calls to dispatch commonTagsHandler. These immediately follow calls to innerExpandMacros which flatten out HERE documents. So a commonTagsHandler should never see unexpanded HERE document syntax.

However that is not the case for all the handlers. The before handlers are all at particular risk of being exposed to HERE documents. Some of these handlers process topic text, for example to detect tables anchored around macros e.g. %TABLE.

The main issue with HERE-document syntax is that it's impossible to guarantee not breaking "old" code, where (unavoidable) assumptions have been made about the rendering order. The same risk applies to any new usage of SECTION blocks, or any other block structuring technique; we are creating blocks where none existed before. There is no magic bullet.

-- CrawfordCurrie - 14 Feb 2010

My problem with *SECTION is that it cannot be queried. Being able to query them would require some more ContentAccessSyntax (wrt to writing queries) anyway. But HERE documents are even more limiting: they can't even be extracted, even if I knew there was one in a particular topic. But headings aren't even on the radar, where there's no syntax at all for working with them, even though this what 99% of our wiki users understand "sections" to be.

I'm probably the only foswiki developer trying to query/mashup topic data that has been structured by ordinary users, without a wiki app developer.

So I'm not proposing any change to this fine proposal.

But it would be nice to still consider extending functionality of SECTIONs as was discussed on the road to working on HERE documents, because I do believe people will want to pass mashable/query-able sections into macros eventually.

-- PaulHarvey - 14 Feb 2010
Topic revision: r9 - 14 Feb 2010, PaulHarvey
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy