(has 3 elements inside)\n"
]
}
],
"source": [
"class CompositeElementIdentificationStep(AbstractProcessingStep):\n",
" def _process(self, elements):\n",
" result = []\n",
" for e in elements:\n",
" if e.html_tag.name == \"div\":\n",
" result.append(\n",
" CompositeSemanticElement.create_from_element(\n",
" e,\n",
" inner_elements=[\n",
" NotYetClassifiedElement(t)\n",
" for t in e.html_tag.get_children()\n",
" ],\n",
" log_origin=\"CompositeElementIdentificationStep\",\n",
" )\n",
" )\n",
" else:\n",
" result.append(e)\n",
" return result\n",
"\n",
"\n",
"parser = Edgar10QParser(lambda: [CompositeElementIdentificationStep()])\n",
"elements = parser.parse(html, unwrap_elements=False)\n",
"show(elements)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We have successfully identified the tag as a `CompositeSemanticElement`.\n",
"\n",
"However, `CompositeSemanticElement` is intended for more advanced use cases, normally we won't even notice it (we had to set `unwrap_elements` flag to `False` to see it):"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"NotYetClassifiedElement
(text: Financial ...)\n",
"NotYetClassifiedElement (text: The financ...)\n",
"NotYetClassifiedElement (text: Strategies...)\n",
"NotYetClassifiedElement
(text: Investment...)\n",
"NotYetClassifiedElement![]()
\n"
]
}
],
"source": [
"elements = parser.parse(html)\n",
"show(elements)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now combine the steps together. One steps output is another steps input, therefore order is important:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MyClassifier: Successfully processed 3 tags!\n",
"\n",
"TitleElement[L0] (text: Financial ...)\n",
"TextElement
(text: The financ...)\n",
"NotYetClassifiedElement (text: Strategies...)\n",
"NotYetClassifiedElement
(text: Investment...)\n",
"NotYetClassifiedElement![]()
\n"
]
}
],
"source": [
"def get_steps():\n",
" return [\n",
" CompositeElementIdentificationStep(),\n",
" MyClassifier(),\n",
" ]\n",
"\n",
"\n",
"parser = Edgar10QParser(get_steps)\n",
"elements = parser.parse(html)\n",
"show(elements)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice that the inner elements of `CompositeSemanticElement` did not get processed. This is because it requires special handling. A simple way to do it would be to inherit from `ElementwiseProcessingStep`:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"from sec_parser.processing_steps import AbstractElementwiseProcessingStep\n",
"\n",
"\n",
"class BetterClassifier(AbstractElementwiseProcessingStep):\n",
" def _process_element(self, element, context):\n",
" if element.html_tag.name == \"b\":\n",
" return TitleElement.create_from_element(element, \"\")\n",
" elif element.html_tag.name == \"p\":\n",
" return TextElement.create_from_element(element, \"\")\n",
" elif element.html_tag.name == \"img\":\n",
" return ImageElement.create_from_element(element, \"\")\n",
" return element"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TitleElement[L0] (text: Financial ...)\n",
"TextElement
(text: The financ...)\n",
"TitleElement[L0] (text: Strategies...)\n",
"TextElement
(text: Investment...)\n",
"ImageElement![]()
\n"
]
}
],
"source": [
"def get_steps():\n",
" return [\n",
" CompositeElementIdentificationStep(),\n",
" BetterClassifier(),\n",
" ]\n",
"\n",
"\n",
"parser = Edgar10QParser(get_steps)\n",
"elements = parser.parse(html)\n",
"show(elements)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We have completed the HTML parsing as the result looks the same as we intended:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TitleElement[L0] (text: Financial ...)\n",
"TextElement
(text: The financ...)\n",
"TitleElement[L0] (text: Strategies...)\n",
"TextElement
(text: Investment...)\n",
"ImageElement![]()
\n"
]
}
],
"source": [
"show(expected_elements)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Introduction to Semantic Trees"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
" Builds a semantic tree from a list of semantic elements.\n",
"\n",
" Why Use a Tree Structure?\n",
" =========================\n",
" Using a tree data structure allows for easier and more robust filtering of sections.\n",
" With a tree, you can select specific branches to filter, making it straightforward\n",
" to identify section boundaries. This approach is more maintainable and robust\n",
" compared to attempting the same operations on a flat list of elements.\n",
"\n",
" Overview:\n",
" =========\n",
" 1. Takes a list of semantic elements.\n",
" 2. Applies nesting rules to these elements.\n",
"\n",
" Customization:\n",
" ==============\n",
" The nesting process is customizable through a list of rules. These rules determine\n",
" how new elements should be nested under existing ones.\n",
"\n",
" Advanced Customization:\n",
" =======================\n",
" You can supply your own set of rules by providing a callable to `get_rules`, which\n",
" should return a list of `AbstractNestingRule` instances.\n",
" \n"
]
}
],
"source": [
"from sec_parser.semantic_tree import TreeBuilder\n",
"\n",
"print(TreeBuilder.__doc__)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"***Let's apply this to our example:***"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"A very similar processing pattern is used here as well:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1;34mTitleElement\u001b[0m: Financial Overview\n",
"└── \u001b[1;34mTextElement\u001b[0m: The financial sector is a categ...ommercial and retail customers.\n",
"\u001b[1;34mTitleElement\u001b[0m: Strategies of Investment\n",
"├── \u001b[1;34mTextElement\u001b[0m: Investment strategies are plans...ign with their financial goals.\n",
"└── \u001b[1;34mImageElement\u001b[0m\n"
]
}
],
"source": [
"from sec_parser.semantic_tree import AlwaysNestAsParentRule, AbstractNestingRule, render\n",
"\n",
"\n",
"def get_rules() -> list[AbstractNestingRule]:\n",
" return [\n",
" AlwaysNestAsParentRule(TitleElement),\n",
" ]\n",
"\n",
"\n",
"builder = TreeBuilder(get_rules)\n",
"tree = builder.build(elements)\n",
"print(render(list(tree)))"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1;34mTitleElement\u001b[0m: Financial Overview\n",
"└── \u001b[1;34mTextElement\u001b[0m: The financial sector is a categ...ommercial and retail customers.\n"
]
}
],
"source": [
"print(render(list(tree)[0]))"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[1;34mTitleElement\u001b[0m: Strategies of Investment\n",
"├── \u001b[1;34mTextElement\u001b[0m: Investment strategies are plans...ign with their financial goals.\n",
"└── \u001b[1;34mImageElement\u001b[0m\n"
]
}
],
"source": [
"print(render(list(tree)[1]))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"For further understanding of `sec-parser`, refer to the [**Documentation**](https://sec-parser.rtfd.io). If you're interested in contributing, consider checking out our [**Contribution Guide**](https://github.com/alphanome-ai/sec-parser/blob/main/CONTRIBUTING.md)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}
![](\"https://en.wikipedia.org/static/images/icons/wikipedia.png\"){kind=icon}
\n",
"